Serving Gemma4 with Rust on vLLM 🦀

This tutorial walks through installing and setting up the Rust toolchain for vLLM on an AWS EC2 G5g instance - Graviton2 (aarch64) with an NVIDIA T4G GPU - and getting vLLM's Rust frontend (vllm-rs ) built, running, and verified. This paper is a follow-on to the original G5g Gemma 4 build. Everything below was run on the box. 🦀 Wait, vLLM has Rust in it? You betcha. Since PR #40848 (merged 2026-05-21), vLLM vendors a 14-crate Rust workspace: bench chat cmd engine-core-client llm managed-engine metrics mock-engine parser parser/python server text tokenizer tracing Edition 2024, resolver 3. Straight from the vendored rust/Cargo.toml : | Crate | Version | Job | |---|---|---| axum | 0.8.8 | the HTTP server | tokio | 1.47.1 | async runtime | zeromq | 0.6.0 | talks to the Python engine | rmp-serde / rmpv | 1.3.1 | msgpack on the wire | minijinja | 2.22 | chat templates | tonic / prost | 0.14.6 / 0.14.3 | gRPC - remember this one | It's a drop-in replacement for the Python FastAPI server. Two artifacts get built: - 🦀 vllm-rs - the axum frontend binary - 🐍 vllm._rust_tool_parser - a PyO3 extension module Rust is a build requirement now That's the headline, and it's reason enough on its own: you cannot build vLLM from source at v0.27.2rc0 without Rust in the picture. setup.py imports it at module scope, line 21, unguarded: from setuptools_rust.build import build_rust No try , no feature flag, no opt-out. Metadata generation doesn't happen without it. And this isn't a quirk of one release. vLLM's Rust surface is 14 crates covering the HTTP frontend, the tool parser, the tokenizer and the benchmark client, and it has been growing since it landed. If you build inference infrastructure from source, a Rust toolchain is becoming table stakes - so it's worth knowing how to drive it properly rather than working around it. Three things do get conflated, though, and they have different scopes: | Component | Needed to build vLLM? | Needed to serve? | |---|---|---| setuptools_rust (Python pkg) | yes, always | no | cargo / rustc toolchain | for working Rust artifacts | no | protoc | for vllm-rs specifically | no | Then why doesn't pip install vllm need this? Because normally pip installs it for you. pyproject.toml declares it: [build-system] requires = [ "cmake>=3.26.1", "ninja", "packaging>=24.2", "setuptools>=77.0.3, =8.0", "setuptools-rust>=1.9.0", # cuda-toolkit-13-2 (sbsa) · rustup 1.97.1 · protobuf-compiler │ │ │ │ HTTP :8000 │ │ | │ │ v │ │ ┌───────────────────────────────┐ │ │ │ [RUST] vllm-rs │ 50 MB aarch64 ELF, its OWN process │ │ │ axum 0.8.8 · tokio │ built from the vendored rust/ workspace │ │ │ minijinja · fastokens │ --host 0.0.0.0 --port 8000 # verify it's really Rust curl -si localhost:8000/health | grep -i '^server:' # Rust sends none pgrep -af vllm-rs Run on EC2 g5g.xlarge and g5g.4xlarge , us-east-1a , NVIDIA T4G (SM 7.5). vLLM 0.27.2rc1.dev0+g7f7a32cfe , rustc 1.97.1, setuptools-rust 1.13.0, libprotoc 3.21.12, torch 2.12.0+cu132. Benchmarks are one run per cell for Rust and two for Python; treat the TPOT delta as suggestive. Top comments (0)