Changing One Prompt Can Affect 50 Others - I Built a Prompt Dependency Graph to Find What Needs Retesting

TL;DR

If you build with composable prompts, changing one shared component can leave you with a difficult question: what actually needs to be re-evaluated?

I built a pure Python prompt dependency graph that answers that question with two numbers:

• Reachable: everything downstream of the changed component-the structural ceiling.
• Candidate: the smaller set that directly depends on the changed section, plus its downstream consumers.

I tested the approach on a deterministic 55-node synthetic system. Depending on how selectively a component is shared, section-aware tracking narrowed the evaluation set by anywhere from 0% to 85% in my experiments.

The important caveat: these numbers identify what should be evaluated, not what will actually fail. Behavioral impact still requires running the evaluation itself.

Results

Here is the data before we get into the code. I ran a 55-node synthetic experiment across a few different change scenarios, tracking the total ceiling versus the actual evaluation set:

The real takeaway here isn't just that a specific change can slash your test burden by nearly half. The catch is how unpredictable the graph is. Sometimes it finds a massive amount of narrowing, and other times it finds nothing. You literally do not know which outcome you are getting until you actually run it.

The Line I Changed That I Couldn't Reason About

I compose most of my production prompts out of shared pieces. There is a base-policy block inherited by a support agent, a sales agent, and an internal analyst agent. There's a tone block that almost imports everything, and a format block controlling JSON versus Markdown output.

It is a completely ordinary setup once you have more than a handful of agents.

One afternoon I changed one sentence in base-policy, extending the refund window from 30 days to 14. The problem wasn't making the edit. It was knowing which agents needed re-evaluation before I shipped it.

Running the entire suite on everything was expensive. I had been guessing which agents used the policy, but I couldn't tell when I was wrong. A correct guess and a lucky one look identical until something breaks. Shipping and waiting for a support agent to quote the old refund window to a real customer was obviously worse than either.

I wanted a structural answer to a simple question: what depends on the thing I just changed?

That question already has a name in traditional software engineering, change impact analysis, which means tracing dependency relationships outward from a change to determine what else needs re-verification [1]. Prompt engineering still lacks many of the lifecycle practices that software engineering takes for granted, and a recent academic proposal for promptware engineering makes the same observation [2].

So I built the smallest version of that I could, and tested it against a system designed specifically to break my assumptions.

Complete code: https://github.com/Emmimal/prompt-dependency-graph/

Why Composability Creates a Larger Evaluation Surface

Every shared component is a single point that many other things depend on. Every edit to it carries a blast radius.

The term comes from explosives, but it has a long second life in software engineering describing how far a change's consequences propagate outward from its source [3]. I am using it in that software-engineering sense here. It means potential downstream impact, not observed behavioral damage. This doesn't measure whether the prompt's output actually changed. It measures how far that change could spread if it did.

Chaos engineering treats minimizing blast radius as a first-class design goal for a simple reason: not to prevent failure, but to keep its consequences bounded and legible [4].

Build systems solve an adjacent problem the same way. Bazel maintains a dependency graph across a codebase specifically so a single file change only triggers rebuilds of what is actually downstream of it, rather than everything [5].

Prompt chaining has its own version of this failure. Because prompts feed into each other, a small and unintended change to an upstream prompt can produce unpredictable results several steps downstream, with no compiler or type system to catch it first [6]. That is exactly what I was worried about with base-policy. I just had no way to measure it.

Why a Flat Dependency Lookup Fails

Before building anything complex, I tried the simplest approach: check who directly imports the component I edited.

I set up two test cases with the exact same total impact (4 downstream nodes), but different layouts:

• Flat: comp-a feeds straight into 4 different agents. (4 direct dependents)
• Deep: comp-b feeds into an agent, which feeds into a workflow, then another agent, and so on. (1 direct dependent)

If you only check direct imports, the flat system reports 4 impacted nodes, but the deep system reports 1. It misses three-quarters of the actual impact.

The reachable downstream set didn't change; both graphs contain 4 downstream nodes. What changed was what a one-hop lookup could see. In the deep case, it finds only the first dependent and misses the three nodes further downstream.

That is the reason transitive analysis matters in traditional change impact analysis: dependencies propagate through chains, so looking only at direct dependents can systematically undercount the downstream evaluation surface [1].

In prompt systems, this happens often because prompts are layered like organizational charts. A base policy goes to an agent, which goes to a workflow, which goes to a router. Every layer is another hop. If a dependency check only looks one level deep, it can miss risks in prompts further down the chain, where important logic often lives.

Component 1: The Data Model

A PromptComponent isn't modeled as an opaque blob of text. It is a versioned collection of named sections.

The section-level approach is the most important design choice because it helps us define "candidate" more narrowly than "reachable."

Agents declare which specific sections they use, and with what relationship (imports, inherits, references, formats-with):

I didn't use dependency type in the impact calculation for v1, but I still needed the model to keep track of the difference. This was a deliberate choice: I made the data model flexible enough for future needs, instead of adding a new field later and changing all the existing dependency declarations. Adding a type system later is harder than including one extra field that you don't use yet.

Component 2: The Dependency Graph

The graph needs to answer two simple questions: who uses this component directly, and who is reachable further down the line?

This dependency walk fixes the blind spot we saw earlier. It treats agent-to-agent and agent-to-workflow connections as normal graph links and uses a simple breadth-first search to follow them outward:

I called this structural_blast_radius in the code, but I just label it Reachable in the output. Saying structural feels a bit too confident since we do not know the full behavioral impact yet. Reachable is a cleaner description of what it actually is: everything downstream that could get touched, rather than everything that definitely will be.

I considered limiting how far the search could go in larger systems, but the results didn't show a need for it. With 55 nodes, the search takes less than a millisecond. Testing it on larger production systems is still future work.

Component 3: Reachable versus Candidate

A section is considered changed when its text is different between two versions. There are no embeddings or semantic checks, just a direct text comparison that ignores extra spaces:

That is one half of the mechanism. The other half runs the same graph traversal, but starts from the specific nodes that declared the changed section instead of the whole component:

That's the whole process. We run the same breadth-first search twice: once for everything connected to the component, and once only for nodes connected to the section that changed.

Defining the Two Numbers

This gives us two clear terms to work with:

Reachable set: The maximum possible impact: every node connected to the changed component, no matter which section changed. It comes only from the graph structure.

Candidate set: The proposed evaluation boundary: nodes that directly depend on the changed section, plus everything downstream from them. It is based on the section diff and graph traversal. Once a node is selected, everything downstream from it is included. For example, if a changed section affects an agent and that agent affects a workflow, the workflow becomes a candidate too. After the first step, we track nodes rather than individual sections.

Important Caveats on the Numbers

Because the reachable ceiling is computed per component rather than per section, it treats every section the same. If a component has one widely used section and one rarely used section, the reachable number stays identical no matter which one you edit. The candidate set is the only metric that actually shifts based on the specific edit.

To keep things precise, I stick to specific terms for these outputs so the tool is never misunderstood:

Component 4: The 55-Node Test System

To test this, I built a deterministic 55-node system consisting of 50 agents across five roles (support, sales, analyst, operations, and marketing, with ten teams each) plus 5 workflow nodes. Those workflow nodes depend on agents rather than components directly, which creates a real transitive and diamond-shaped dependency structure.

There are also five shared components (base-policy, tone, format, domain, safety), each with two to four sections.

The key rules behind this table are simple: support imports the refunds and privacy sections, sales only imports privacy, and marketing doesn't touch the base policy at all. That last rule is deliberate. If every node in a synthetic system depended on every shared component, it would hide the exact pattern I wanted to find. You need some agents with zero exposure to a given component to test whether section-aware tracking actually works.

The full generator covering all five roles and the workflow layer lives in the repository. I left out the 50 near-identical configuration blocks here to save space.

Every number in the results table at the top of this article comes from running compute_impact() against this exact graph. These figures describe one deliberately built, synthetic 55-node system rather than a universal law for every production suite. Whether a real system shows this same pattern depends entirely on its actual dependency structure, which is what this tool exists to measure rather than assume.

Walking Through One Change, Start to Finish

With the system in place, it is worth tracing what happens during a real edit because the mechanism is easy to lose track of in the abstract.

Say I edit the refunds section of base-policy:

v1: "Customers may request refunds within 30 days of purchase."
v2: "Customers may request refunds within 14 days of purchase."

changed_sections() compares every section of v1 against v2. It sees that refunds differs, while privacy and escalation do not.

Next, structural_blast_radius("base-policy") walks the graph outward through every node depending on it for any reason, plus everything downstream of those. That gives us 45 nodes.

Meanwhile, section_dependents("base-policy", "refunds") narrows that list down to only the nodes that explicitly declared a dependency on refunds, extended downstream. That gives us 24 nodes.

Why the Difference Matters

The gap between those two sets is concrete. Every sales-* agent is in the reachable set of 45 because they depend on base-policy through the privacy section. However, none of them are in the candidate set of 24 because they never declared a dependency on refunds.

A sales agent uses the privacy section because it handles customer data, but it does not use the refund section. Without section-level declarations, we cannot tell the difference.