Running Gemma 4 26B on a 13-Year-Old Xeon: Practical AI Performance Without GPUs

Originally published on tamiz.pro.

Introduction

Large language models like Gemma 4 26B typically require powerful GPUs with high VRAM. This tutorial demonstrates how to run the model on a 13-year-old Xeon processor (e.g., Intel Xeon E5 v2 series) using CPU-only optimization techniques like model quantization and memory-efficient execution.

Prerequisites

• Xeon-based server with at least 64GB RAM
• Linux OS (Ubuntu/CentOS recommended)
• Python 3.10+
• git, cmake, and gcc installed
• At least 200GB free disk space

Step 1: Prepare the Environment

Start by installing core dependencies:

sudo apt-get update
sudo apt-get install -y python3-pip build-essential
pip install torch==2.1.0 transformers optimum

Verify PyTorch's CPU support with:

import torch
print(torch.__version__, torch.cuda.is_available())  # Should return False

Step 2: Download and Convert the Model

Use Hugging Face's from_pretrained with quantization:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "google/gemma-4-26b"
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Load with 4-bit quantization
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_4bit=True,
    torch_dtype=torch.float16
)

This reduces RAM usage from 120GB (float16) to ~40GB through quantization.

Step 3: Optimize Memory Usage

Add CPU-specific optimizations:

from torch._dynamo import optimize_for_cpu

# Enable optimized CPU execution
model = optimize_for_cpu(model)
model.tie_weights()

# Configure attention computation
import torch.nn as nn
nn.Linear(model.config.hidden_size, model.config.hidden_size).to(memory_format=torch.channels_last)

Step 4: Run Inference

Execute with batch size 1 and CPU-optimized pipeline:

input_text = "Explain quantum computing in simple terms"
inputs = tokenizer(input_text, return_tensors="pt")

# Use CPU for inference
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=100)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Performance Expectations

Optimization Tips

• Use load_in_8bit instead of load_in_4bit if RAM is constrained
• Disable gradient computation globally
• Use --cpu-inference flag in any training scripts
• Enable Intel MKL optimizations:

export MKL_THREADING_LAYER=GNU
export MKL_SERVICE_FORCE_INTEL=1

Conclusion

While modern GPUs provide better throughput (100-300 tokens/sec), this CPU-only approach enables AI inference on legacy hardware at ~15% of GPU costs. Ideal for edge deployments or proof-of-concept work. Consider upgrading to Xeon Scalable (2nd Gen) for production workloads requiring higher throughput.