Hacker News user runs Google Gemma 4 26B on legacy Xeon hardware without GPU
A community-driven technical patch enables the 26-billion-parameter mixture-of-experts model to operate at approximately five tokens per second on pre-AVX2 Intel processors, challenging assumptions about modern AI infrastructure costs.
A user on Hacker News has successfully deployed Google’s Gemma 4 26B mixture-of-experts language model on a 13-year-old HP StoreVirtual storage appliance, operating entirely without a graphics processing unit. The system, equipped with dual Intel Xeon E5-2690 v2 processors, achieved a generation speed of approximately five tokens per second. This effort highlights the potential for repurposing older enterprise hardware for AI inference tasks, provided specific software compatibility issues are resolved.
The experiment utilised the ik_llama.cpp inference engine, a fork of llama.cpp that adds optimisations for Gemma 4’s mixture-of-experts architecture. However, the user encountered immediate compatibility barriers because the engine’s fast kernels assume the AVX2 instruction set, which was introduced in the Haswell architecture (v3 Xeons) in 2014. The user’s Ivy Bridge-based hardware (v2 Xeons) predates this instruction set, causing the build to fail on startup due to missing optimized paths.
To resolve these issues, the user collaborated with the Claude AI assistant to diagnose and patch the code. The collaboration involved modifying the inference engine to bypass AVX2 requirements, specifically by splitting fused operations into separate calls that allow the code to fall back to scalar and SSE math on older silicon. The technical patch was submitted as a pull request to the ik_llama.cpp repository and is currently awaiting maintainer review.
The debugging process revealed that the model was producing fluent but nonsensical output because certain graph operations were falling through to a default case that left hidden state tensors uninitialized. The AI assistant identified this by instrumenting raw logits, noting a mean logit of plus 16 which indicated uninitialized memory rather than random corruption. The resulting fix ensures that the model runs correctly on pre-AVX2 chips without sacrificing the performance benefits of the fork on newer hardware.
In addition to the instruction set mismatch, the user identified a separate bug related to the `--run-time-repack` flag, which reorders weights into an AVX2-only layout. The workaround involves dropping this flag from the run script. The user, who noted they are not a C++ programmer, drove the process by running experiments and interpreting outputs, while the AI assistant handled the complex code modifications.
The successful deployment offers a low-cost alternative to cloud-based AI subscriptions. The user estimated the server cost under $300, positioning it as a viable fallback for when paid APIs are unavailable or for grinding through slow batch jobs where paying per token is inefficient. The experiment underscores the value of understanding model internals and legacy system constraints to extend the utility of older hardware.
