Context Assembly Is a Build Step: Versioning the Prompt You Actually Send

You review a 30-line template diff. Production sends 40,000 tokens assembled at request time from six sources, four of which changed this week and none of which appear in the pull request. The diff you approved is not the change you shipped. This is the part of LLMOps that keeps biting teams after they have evals, tracing, and a gateway in place. The template is versioned in git. The context is assembled by code paths nobody treats as release artifacts. The prompt is a build output List what actually goes into a request for a typical RAG-plus-tools system: - The system template (in git, reviewed). - Top-k retrieved chunks from an index that gets rebuilt on its own schedule. - Tool JSON schemas, whose descriptions are prompt text written by whoever last touched the tool. - A conversation summary produced by another model call. - A user profile or tenant config blob. - Few-shot examples chosen at runtime by a selector. Each is a dependency with its own version and its own deploy cadence. The retrieved chunks change when someone reindexes. The tool schema changes when a backend engineer renames a parameter and rewrites the description. Neither triggers a review of the prompt. So when quality drops on Tuesday and no code shipped, there is no diff to look at. That is the failure this whole post is about. Fingerprint everything the model sees Compute a content hash over the inputs that can change model behaviour, and log it with every trace: - model id and version - template version - retrieval snapshot id (the index build, not the query) - the fully rendered prompt - decoding parameters that matter Store fingerprints always. Store full rendered prompts sampled or redacted, depending on your data rules - full-text retention is a compliance decision, not an observability one. The fingerprint is small, cheap, and answers "what changed" during an incident in one query. The trade-off: a fingerprint that includes a timestamp, a request id, or a per-user greeting has near-zero collision rate and zero diagnostic value. Hash the parts that affect generation, not the metadata around it. I have watched a team spend a day on "why is our prompt hash changing every request" when the answer was a locale string. A small assembler with provenance and eviction You do not need a framework for this. You need three properties: every section carries a trust level, every section carries an eviction priority, and over-budget on a pinned section is a hard error rather than a silent trim. from dataclasses import dataclass, field from hashlib import sha256 import re @dataclass class Section: name: str text: str trusted: bool # False for retrieval, tool output, user uploads priority: int # higher survives truncation pinned: bool = False # never evict; over budget means fail loudly INSTRUCTION_LIKE = re.compile( r"(ignore (all )?(previous|prior) instructions|you are now|system prompt)", re.IGNORECASE, ) def screen(text: str) -> str: # Cheap heuristic, not a guarantee. Log hits and route to a stricter path. return INSTRUCTION_LIKE.sub("[filtered]", text) def count_tokens(s: str) -> int: # Swap in tiktoken/the provider counter. One counter per model family. return len(s) // 4 @dataclass class Assembled: prompt: str fingerprint: str dropped: list[str] = field(default_factory=list) def assemble(sections, budget, model_id, template_version, snapshot_id): ordered = sorted(sections, key=lambda s: (s.pinned, s.priority), reverse=True) kept, dropped, used = [], [], 0 for s in ordered: body = f" \n" body += (s.text if s.trusted else screen(s.text)) + f"\n " cost = count_tokens(body) if used + cost > budget: if s.pinned: raise ValueError(f"pinned section {s.name} exceeds budget") dropped.append(s.name) continue kept.append((s, body)) used += cost # Render in original order so layout is stable across requests. pos = {id(s): i for i, s in enumerate(sections)} kept.sort(key=lambda pair: pos[id(pair[0])]) prompt = "\n\n".join(body for _, body in kept) h = sha256() for part in (model_id, template_version, snapshot_id, prompt): h.update(part.encode()) return Assembled(prompt, h.hexdigest()[:16], dropped) # Usage: build once per request, log the fingerprint beside the trace id. ctx = assemble( sections=[ Section("question", "why was my claim denied?", trusted=True, priority=10, pinned=True), Section("policy_chunk", retrieved_text, trusted=False, priority=5), Section("profile", tenant_blob, trusted=True, priority=1), ], budget=12_000, model_id="claude-sonnet-4", template_version="support-v14", snapshot_id=index_build_id, ) Two details in there are load-bearing. Sections render in original insertion order rather than priority order, because reordering the same content changes model behaviour and makes your eval deltas unattributable. And pinned turns budget pressure into an exception instead of a degraded Top comments (0)