From API to GPU, Week 2: What Actually Happens Behind the API

Phase 1 of 8: Comfortable running local models. Week 2 of 32.

In Week 1 I went through the machine and proved the GPU works. This week I run an actual language model on it and, for the first time, look at what sits behind the "AI API" I have called for years. The short version: the API is not magic. It is an HTTP layer over a local process that loads a file of numbers into memory and does the matrix math from Week 1. By the end of this post that sentence will be clear, backed by real commands and real output you can reproduce.

I am using Ollama, a runtime that makes running a local model about as easy as running a container. Other names you will see in this space are llama.cpp for lightweight local inference, vLLM for high-throughput serving, and NVIDIA TensorRT-LLM for optimized NVIDIA inference. Hugging Face Transformers is also common, but it is a broader Python framework for running and training models, not a ready-made local model service. These tools overlap, but they are not exact replacements.

I chose Ollama because it gives me a CLI and local HTTP API with very little setup. 1 Everything below runs on the DGX Spark and is reached over SSH as spark, the same setup as Week 1.

Concept 1: a model is not a runtime

The first idea to keep straight, because it holds for the rest of the series, is the split between the model and the runtime. The model is trained data: weights plus the metadata needed to use them. On its own it does nothing. It is data on disk. The runtime is the program that loads those weights into memory and runs the math to turn your prompt into text.

Ollama is the runtime here. If you know Docker, there is a useful analogy, but it is not exact. A model package is like an image made of versioned, content-addressed layers. Ollama is like the engine that pulls those layers and starts the workload. Unlike a container image, the main model layer is trained numeric data, not an app and its operating-system files.

Ollama calls each stored package file a blob. Here, a blob is just a file kept under a name derived from its content digest. The inspection below follows the package index to the model blob and checks that it is the expected file.

Rather than run a wrapper script, I inspected the model with a few direct commands, one at a time. Each command answers a single question, so you can paste it, read the output, then move to the next.

First, list what Ollama has downloaded locally:

ssh spark 'ollama list'

NAME                          ID              SIZE      MODIFIED
nomic-embed-text:latest       0a109f422b47    274 MB    2 days ago
llama3.2:3b                   a80c4f17acd5    2.0 GB    3 days ago
phi4:latest                   ac896e5b8b34    9.1 GB    7 days ago
qwen3.6:35b-a3b-bf16         94061ddd23a7    71 GB     8 days ago

Next, ask Ollama for the focused facts about Phi-4. The Ollama server listens on localhost:11434 on the Spark, so run this in a shell on the Spark (ssh spark):

curl -s http://localhost:11434/api/show -d '{"model": "phi4"}' | jq '{
  format: .details.format,
  architecture: .details.family,
  parameters: .details.parameter_size,
  context_length: (.model_info | to_entries | map(select(.key | endswith(".context_length"))) | first.value),
  embedding_length: (.model_info | to_entries | map(select(.key | endswith(".embedding_length"))) | first.value),
  quantization: .details.quantization_level,
  capabilities,
  runtime_parameters: (.parameters | split("\n") | map(select(length > 0) | gsub(" +"; " ")))
}'

{
  "format": "gguf",
  "architecture": "phi3",
  "parameters": "14.7B",
  "context_length": 16384,
  "embedding_length": 5120,
  "quantization": "Q4_K_M",
  "capabilities": ["completion"],
  "runtime_parameters": [
    "stop \"<|im_start|>\"",
    "stop \"<|im_end|>\"",
    "stop \"<|im_sep|>\""
  ]
}

That call gives readable model facts, including "format": "gguf". For a normal check, this is how I know Ollama identifies the model as GGUF. I do not need to locate the raw file and inspect its bytes just to answer that question.

To see the exact files that make up the package, read Ollama's manifest. It is a small JSON file on the Spark, so jq can read it directly. I select just the package version and the file list:

jq '{schemaVersion, layers: [.layers[] | {mediaType, digest, size}]}' \
  /usr/share/ollama/.ollama/models/manifests/registry.ollama.ai/library/phi4/latest

{
  "schemaVersion": 2,
  "layers": [
    {
      "mediaType": "application/vnd.ollama.image.model",
      "digest": "sha256:fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20",
      "size": 9053114464
    },
    {
      "mediaType": "application/vnd.ollama.image.template",
      "digest": "sha256:32695b892af87ef8fca6e13a1a31c67c1441d7398be037e366e2fc763857c06a",
      "size": 275
    },
    {
      "mediaType": "application/vnd.ollama.image.license",
      "digest": "sha256:fa8235e5b48faca34e3ca98cf4f694ef08bd216d28b58071a1f85b1d50cb814d",
      "size": 1084
    },
    {
      "mediaType": "application/vnd.ollama.image.params",
      "digest": "sha256:45a1c652dddc9efdcefa977ab81cfbe26b6e52bc8e78f2f4c698538783e0ac80",
      "size": 82
    }
  ]
}

The registry path ends in library/phi4/latest, so this manifest is the index for the model name phi4 and tag latest. It does not hold the 9.1 GB of weights. It lists the files that make up the Ollama package, much like a lock file maps package names to exact artifacts.

Each entry under layers is one file in the package (one blob). It has three useful fields:

• mediaType says what the file’s role is. This package has a model, a prompt template, a license, and default parameters.
• digest is the file’s content identity. The sha256 prefix names the hash algorithm, and the characters after the colon are the hash of the file’s bytes.
• size says exactly how many bytes that file should contain.

The model layer’s digest is also how Ollama names the file on disk. It stores each blob under a blobs directory using the digest as the filename, with the colon changed to a hyphen, so sha256:fd7b...df20 becomes sha256-fd7b...df20. That file is 9,053,114,464 bytes, the size shown above, and its SHA-256 matches the digest. The linked investigation independently checks the file header with xxd: bytes 47 47 55 46 spell GGUF in ASCII. That deeper check is useful when reusing the raw file in another runtime, but .details.format is the simple Ollama API answer. 2

So ollama show and the manifest answer different questions. The metadata call gives readable model facts such as architecture and context length. The manifest tells Ollama which exact files form the runnable package and where to find them by content ID.

You might now wonder whether another runtime can use the same GGUF file. That is useful, but it is not part of the main Ollama lesson. I answer it near the end in Can llama.cpp reuse this Ollama model?.

I am using phi4, Microsoft's 14.7B-parameter model. At 16 bits per weight, 14.7 billion weights alone would need about 29.4 GB. The local package is only 9.1 GB, which leads to the first surprise. A quick tour of what each line means, since these terms come up constantly:

That last row answers the 9.1 GB surprise. This build is quantized: its weights use a mixed low-bit Q4_K_M representation instead of 16-bit values. Four bits per weight would be about 7.35 GB before metadata and quantization overhead, so a 9.1 GB model layer is reasonable. Quantization is a whole phase later in this series (weeks 14 to 16). For now the only thing to take away is that Ollama runs lower-precision weights, and that is why this 14.7B model takes much less space than its 16-bit source model.

The three stop entries (<|im_start|>, <|im_end|>, <|im_sep|>) are special tokens: markers the model was trained to use as turn boundaries in a chat. The runtime watches for them to know when the model has finished its reply. They show up again in the next section, so keep them in mind.

Concept 2: the API is just HTTP over a local process

Ollama exposes a local HTTP API on port 11434, 3 the same shape of thing as any cloud AI API. The difference is that the "server" is a process on my own machine. I called the generate endpoint with curl and formatted the reply with jq. Run it directly on the Spark:

curl -s http://localhost:11434/api/generate -d '{
  "model": "phi4",
  "prompt": "Explain Kubernetes scheduling in three sentences.",
  "stream": false
}' | jq

Here is the complete response, nothing removed. It is long, but seeing all of it is the point, because two parts of it explain how the model actually works:

{
  "model": "phi4",
  "created_at": "2026-07-14T03:11:50.862545331Z",
  "response": "Kubernetes scheduling is the process of assigning workloads, such as pods, to appropriate nodes within a cluster based on resource availability and constraints. The scheduler evaluates factors like CPU, memory requirements, affinity/anti-affinity rules, taints/tolerations, and node selectors to determine the best fit for each pod. This ensures optimal utilization of resources, workload distribution, and adherence to specified policies, thereby maintaining efficient cluster operation and performance.",
  "done": true,
  "done_reason": "stop",
  "context": [
    100264, 882, 100266, 198, 849, 21435, 67474, 38952, 304, 2380,
    23719, 13, 100265, 198, 100264, 78191, 100266, 198, 42, 30927,
    38952, 374, 279, 1920, 315, 61853, 990, 33785, 11, 1778, 439,
    55687, 11, 8475, 7954, 2949, 264, 10879, 3196, 389, 5211, 18539,
    323, 17413, 13, 578, 29909, 67349, 9547, 1093, 14266, 11, 5044,
    8670, 11, 51552, 14, 15719, 71260, 13797, 5718, 11, 259, 1673,
    82, 5640, 22847, 811, 11, 323, 2494, 57137, 311, 8417, 279,
    1888, 5052, 369, 1855, 7661, 13, 1115, 26420, 23669, 50549,
    315, 5070, 11, 54696, 8141, 11, 323, 71628, 311, 5300, 10396,
    11, 28592, 20958, 11297, 10879, 5784, 323, 5178, 13
  ],
  "total_duration": 3974942694,
  "load_duration": 100037676,
  "prompt_eval_count": 18,
  "prompt_eval_duration": 46132000,
  "eval_count": 89,
  "eval_duration": 3817895000
}

Reading it top to bottom:

• The response field is the answer, a real reply from a model running on my hardware with no network call leaving the box.
• The context field is the part worth staring at. You might be wondering what those numbers are. They are tokens. A model does not read text; it reads token IDs, which are integers. Before anything runs, a tokenizer splits the text into tokens and maps each one to an integer. And this array is not just the prompt, it is the whole conversation as tokens: the chat template, my question, and the model's full answer.

Decoding the first several so it is not a mystery:

• 100264 is the special marker <|im_start|> (start of a turn), and 882 is the word "user". So the conversation begins "start of turn, user".
• 849, 21435, 67474, 38952 are "Ex", "plain", " Kubernetes", " scheduling". Notice "Explain" is split into two tokens. Tokens are often sub-word pieces, not whole words.
• 304, 2380, 23719, 13 are " in", " three", " sentences", ".", finishing my prompt.
• 100265 is <|im_end|> (end of turn), then 100264 78191 100266 is the start of the assistant's turn ("start, assistant, separator").
• 42, 30927 onward ("K", "ubernetes", ...) is the model's answer, token by token, which is exactly the text in the response field above.

The special IDs 100264, 100265, 100266 are the chat markers (<|im_start|>, <|im_end|>, <|im_sep|>), the turn boundaries from the model's embedded tokenizer and chat template.

Two questions probably popped into your head reading that, the same ones that popped into mine: wait, how do I even know 849 is "Ex" or that 100264 is <|im_start|>? And how does Ollama know which tokenizer to use? Both have clear answers, but they would break the flow here, so I answer them at the end. If you want them now, jump to the token questions, answered.

This is why token counts matter everywhere. On a cloud API you pay per token. In a model, tokens are what fill the context window, and the whole conversation is carried forward as this growing list of integers. When people say "tokens per second," this array is the unit being counted.

Concept 3: inference has two measured phases

The bottom of that same output has the timing, and it turns vague words like "latency" into measured numbers. All the durations are in nanoseconds, so here they are converted:

• prompt_eval_duration is the time spent reading the tokenized input. That is the prefill phase.
• eval_duration is the time spent generating output tokens. That is the decode phase.
• load_duration is time Ollama spent loading or preparing the model for this request.

One headline number can be calculated directly: Tokens per second (generation speed): 89 tokens divided by 3.82 s is about 23.3 tokens/sec.

I cannot honestly calculate time to first token (TTFT) from this non-streaming response. Adding load and prefill gives 146 ms of server work before decode, but that is not the same as observing when the first token reaches the client. To measure TTFT, the client must request a streamed response and timestamp the first non-empty token. I do that in the benchmark later in this post.

The timing split also shows that generating text has two phases that behave differently:

• Prefill (prompt_eval): the model reads all 18 prompt tokens at once. This is fast (46 ms) because the tokens are processed together, which is the parallel, compute-heavy work the GPU from Week 1 is good at.
• **Decode (eva