Deploying the 600GB Inkling-NVFP4 Model on Spot A3: A GKE and vLLM Deep Dive

Ok, so, maybe you're a software developer or data scientist who just heard about the new, massive 600GB Inkling-NVFP4 AI model, and you want to try running it yourself without breaking the bank. You also might have learned through some research that renting a "Spot A3" instance on Google Kubernetes Engine (GKE) is a brilliant way to do it. An A3 instance is essentially a massive supercomputer packed with 8 top-tier NVIDIA H100 GPUs, and "Spot" means we are renting it at a huge discount. You set up your cluster, point it to your AI engine, hit deploy, and... it crashes.. But what exactly is causing that? Why does our shiny new AI supercomputer crash before we even send it a single prompt? Deploying cutting-edge, massive models introduces a perfect storm of software conflicts and physical memory limits. At a high level, this is how you navigate these hurdles to get your AI engine running smoothly without writing hacky installation scripts or building your own custom Docker images. Software Conflicts: Swapping to the Official vLLM Image The first major hurdle most people hit is a software conflict (specifically, an ABI mismatch). Deploying a model like Inkling requires gigabytes of bleeding-edge AI software (like vllm 0.25+ and transformers ). If you try to install these new AI packages on top of the default Ray cluster image (rayproject/ray), they will automatically upgrade underlying math libraries like numpy to version 2.0. Because the cluster's base system was compiled against numpy version 1.0, this causes a severe communication breakdown. Imagine two high-speed construction foremen coordinating across a busy site. One foreman is speaking a legacy 1.0 dialect, while the other suddenly switches mid-shift to a brand-new 2.0 slang. When they try to send messages back and forth across the cluster nodes, they literally lose the ability to speak the same language and crash instantly. Furthermore, if you try to install these heavy packages on the fly when the server starts up, you will almost certainly hit Ray's built-in 10-minute download timeout limit. It is a tricky one to debug when you are first starting out, and a quite noticeable performance hit to developer velocity. To cleanly solve this without building a custom Docker container, you should simply use the official vLLM image as your cluster's base image. The official vllm/vllm-openai:v0.26.0 image already has ray, vllm, transformers, and the correct version of numpy beautifully packaged together. By swapping out your Kubernetes YAML manifest to use this image for both the Head and Worker nodes, the servers boot up instantly with no API conflicts, no ABI mismatches, and zero download timeouts. Memory Constraints: Squeezing the KV Cache into VRAM Once the software environment is running smoothly, we hit the hard physical limits of the hardware itself. An A3 instance gives us 8 NVIDIA H100 GPUs, providing a massive 640GB of total High Bandwidth Memory (VRAM). However, the Inkling-NVFP4 model we want to run is roughly 600GB. This means the model weights alone consume almost all the available space on the cards. When the AI engine boots up, it tries to pre-allocate space for something called the KV Cache. The KV Cache is basically the AI's short-term memory that it uses while generating a sentence. By default, Inkling wants to remember up to 1 million words at a time. Trying to reserve enough short-term memory for 1 million words requires about 7GB of space per GPU, which immediately causes an Out-Of-Memory (OOM) crash because our graphics cards are already 95% full! To visualize this memory squeeze, imagine trying to pack a massive grand piano into a delivery van. The piano takes up 95% of the cargo space. If you then try to cram ten giant moving trunks (a 1-million-token KV cache) into the tiny remaining sliver of space, the doors won't shut and the axle snaps. To squeeze into the remaining memory safely, we tweak two critical engine arguments to swap those giant moving trunks for a compact briefcase: - Set max_model_len=4096 : This limits the AI's short-term context window to 4,000 words instead of 1 million, drastically reducing the space needed down to a few megabytes per GPU. - Set gpu_memory_utilization=0.96 : This tells the vLLM engine it is only allowed to use 96% of the graphics card memory, making sure it leaves a couple of gigabytes (about 4% headroom) for the operating system and CUDA runtime to function without crashing. Connecting the Dots: Clean Python Serving Because we used the official vLLM image for our cluster, our deployment code doesn't need any messy installation hacks or shell scripts. Here is the beautifully clean Python script using Ray Serve to configure the memory limits and serve the model across all 8 GPUs: import ray from ray import serve from fastapi import FastAPI, Request from fastapi.responses import JSONResponse # Connect to the Ray cluster ray.init(address="auto") app = FastAPI() # Tell Ray Serve we need 8 GPUs to run this deployment actor @serve.deployment( num_replicas=1, ray_actor_options={"num_gpus": 8} ) @serve.ingress(app) class InklingDeployment: def init(self): from vllm.engine.arg_utils import AsyncEngineArgs from vllm.engine.async_llm_engine import AsyncLLMEngine # Configure the vLLM inference engine engine_args = AsyncEngineArgs( model="thinkingmachines/Inkling-NVFP4", tensor_parallel_size=8, # Split the model weights across all 8 GPUs gpu_memory_utilization=0.96, # Leave 4% VRAM headroom for the OS and CUDA max_model_len=4096, # Cap short-term memory to 4k tokens to fit VRAM trust_remote_code=True, enforce_eager=True ) self.engine = AsyncLLMEngine.from_engine_args(engine_args) # Health check endpoint to confirm the model is online @app.get("/health") async def health(self): return {"status": "Inkling-NVFP4 is ready!"} # Generation endpoint to handle incoming prompts @app.post("/completions") async def generate(self, request: Request): from vllm import SamplingParams from vllm.utils import random_uuid request_dict = await request.json() prompt = request_dict.pop("prompt") sampling_params = SamplingParams(**request_dict) results_generator = self.engine.generate(prompt, sampling_params, random_uuid()) final_output = None async for request_output in results_generator: final_output = request_output return JSONResponse({"text": final_output.outputs[0].text}) if name == "main": serve.start(detached=True) serve.run(InklingDeployment.bind(), route_prefix="/inkling") Note here that we specify @serve.ingress(app) right at the top of the class definition. It is a good habit to be getting into to ensure the Ray cluster properly routes your HTTP endpoints (like /health and /generate) to external traffic. Resources & Next Steps So, now you've got your serving infrastructure configured, your 600GB Inkling-NVFP4 model slotting into VRAM alongside the KV cache on your GKE Spot A3 node pool, and you can successfully chat with your massive AI model without breaking the bank. If you want to dive deeper into the model weights, engine options, or cluster orchestration used in this deep dive, check out these official documentation pages and resources: - Thinking Machines Inkling-NVFP4 on Hugging Face: https://huggingface.co/thinkingmachines/Inkling-NVFP4: Download model weights, review model card specs, and check tokenizer settings. - vLLM Documentation: https://docs.vllm.ai/: Explore vLLM engine configurations, PagedAttention memory management, and tensor parallelism flags. - Ray Serve Documentation: https://docs.ray.io/en/latest/serve/index.html: Learn more about Ray Serve deployments, multi-GPU actor scheduling, and HTTP ingress routing on Kubernetes. Let us know how your GKE Spot A3 deployment went, and happy serving! Top comments (0)