You Recorded Every Event. Can You Still Reconstruct the Execution?

You Recorded Every Event. Can You Still Reconstruct the Execution?

Imagine that, months after an AI workflow ran, you want to reconstruct exactly what happened. The customer asked for a research report. The report was eventually delivered, and your system recorded every relevant runtime event along the way. Nothing is obviously missing.

You have events for planning, search, tool calls, model calls, validation and the final successful result. Each record has a timestamp. The provider calls are there. The failures are there too.

Your data might look something like this:

10:00:01  request accepted
10:00:03  planning started
10:00:07  search
10:00:08  search
10:00:12  tool call
10:00:15  tool call failed
10:00:18  tool call
10:00:26  model call
10:00:31  validation failed
10:00:37  model call
10:00:44  validation succeeded
10:00:45  report delivered

At first glance, this looks like a pretty good execution history. But try answering a few questions from it:

• Was the second tool call a retry of the failed one, or a different operation?
• Did the two searches run sequentially or as parallel branches?
• Was the second model call a retry, a fallback, or a new stage of the workflow?
• And if some of this work continued asynchronously, did it still belong to the execution initiated by the original request?

The individual events may all be accurate. The structure that connected them may already be gone. That's the part I've been thinking about while working on AI monetization infrastructure.

It's tempting to treat reconstruction as an aggregation problem: preserve the events now, group them later, and the execution history will still be there when you need it. I'm becoming less convinced that this is enough.

A distributed workflow is not only a collection of things that happened. It also contains relationships: one operation spawned another, two operations ran as siblings, an attempt retried an earlier attempt, a worker continued work after the original request ended, or several branches eventually contributed to the same result. If those relationships disappear, having every event does not necessarily give us the execution back.

We didn't lose the events. We lost the relationships between them.

That suggests a different engineering question. Not only: What events should an AI runtime preserve? But: What identity and lineage must survive if we want to reconstruct how those events belonged together?

A Request Has a Lifetime. The Execution May Have Another.

HTTP gives us a very convenient mental model:

request ↓ work ↓ response

For simple synchronous operations, that model can also provide a useful boundary for observability. A request arrives, the application performs some work, returns a response, and much of what we care about happens within that lifetime.

Distributed AI workflows can break that assumption very quickly.

Suppose generating the research report takes long enough that we don't want the client to keep an HTTP connection open. The API accepts the request, creates some work and returns 202 Accepted.

                    HTTP request
                         ↓
                      accepted
                         ↓
                     enqueue job
                         ↓
                    202 Accepted
                         │
                         │  execution continues
                         ↓
                      worker
                         ↓
                     planning
                         ↓
                     fan-out
                    /   |   \
                  /     |     \
              search  search   tool
                 A      B       C
                 ↓       \    /  |
              retry       \|/    \
                         \|/      \
                       aggregation
                          ↓
                       validation
                          ↓
                        outcome

The request may have lived for a few hundred milliseconds. The work it initiated may live for minutes. That difference matters because a request_id can still correctly identify the interaction that entered the system without necessarily being the right identity for everything that happens afterward.

The queue message may be delivered later. A worker may create several child jobs. One branch may retry independently. Another may call an external service and wait for a callback. The workflow may pause and resume after the original process that handled the HTTP request no longer exists.

We can propagate the original request context through those boundaries, and doing so is extremely useful. Distributed tracing is specifically designed to carry context across services and process boundaries so related operations can remain observable as part of a distributed flow.

But propagation does not make the original request and the resulting domain execution the same concept. Consider:

req_123
  ↓
execution_123
  │
  ├── search_job_A
  ├── search_job_B
  └── tool_job_C
        ↓
      FAILED
        ↓
      retry

The request tells us where this interaction entered the system. What we're trying to reconstruct later is something slightly different: the logical work that continued because of it.

That distinction becomes more important when work can resume without a new customer request, when one request initiates multiple independent executions, or when a later callback continues an execution that started somewhere else.

So I don't think the useful conclusion is simply: Replace request IDs with execution IDs. We still want request identity. It answers a real operational question. The problem is assuming that one identifier can represent every kind of identity we care about. A request belongs to the transport interaction. An execution may need to survive beyond that interaction.

And once an execution can survive its request, another problem appears almost immediately. What happens when the same logical work is attempted more than once?

One Logical Execution Can Leave More Than One Attempt Behind

Retries make the identity problem harder because one piece of logical work can produce multiple physical attempts. Suppose one branch of our research workflow calls an external tool:

execution_123
  ↓
tool_call
  ↓
attempt_01
  ↓
provider
  ↓
timeout

From our side, the call timed out. We don't know whether the provider rejected it, started processing it, completed it but failed to return the response, or consumed resources before something else went wrong.

So we retry:

execution_123
  ↓
tool_call
  │
  ├── attempt_01
  │     ↓
  │   timeout
  │
  └── attempt_02
        ↓
      success

From the workflow's perspective, this may still be one logical operation: call the tool and obtain a result. From the runtime's perspective, two attempts happened.

That difference matters because several questions that look similar are actually independent:

• Did the retry produce duplicated application state? An idempotency mechanism may help us prevent or detect that.
• Did the provider execute the first attempt despite our timeout? That depends on evidence we may or may not have.
• Did both attempts consume billable resources? That's another question.
• And if both consumed resources, should both eventually be associated with the execution that produced the customer outcome? That's an attribution question we haven't answered yet.

Idempotency is particularly easy to overextend conceptually. An idempotency key can help a system recognize repeated processing of the same logical operation and avoid applying the same effect more than intended. But that does not mean only one physical attempt occurred. A retry can be safe from the perspective of application state while still representing additional runtime work.

For reconstruction, preserving only the final logical state can therefore hide something important:

logical operation
  ↓
SUCCESS

Compare that with:

logical operation
  │
  ├── attempt_01
  │     ↓
  │   timeout
  │
  └── attempt_02
        ↓
      SUCCESS

Both representations may describe the same final application state. They do not preserve the same execution history.

This is why I find it useful to distinguish, at least conceptually, between an execution identity and an attempt identity. The execution identifies the logical work we're trying to follow. The attempt distinguishes a particular try at performing some part of that work. I don't think every system needs those exact names or separate persisted identifiers for every operation. The useful distinction is semantic, not terminological.

If multiple attempts can happen and those attempts matter to questions we may ask later, the runtime needs some way to preserve that relationship. Otherwise:

tool_call   timeout
tool_call   success

leaves us trying to infer whether we observed a retry, two independent operations, or something else entirely.

And retries are still the easy shape. Once one execution starts creating several pieces of work in parallel, a flat sequence of individually accurate events becomes even less representative of what actually happened.

A Flat Event List Can Lose the Execution Structure

Fan-out makes the problem more obvious. Our research workflow might reach a planning stage and perform several operations in parallel:

execution_123
  ↓
planning
  ↓
fan-out
 / | \
/  |  \
search  search  tool
  A      B      C
  \      |     /
   \     |    /
    aggregation
       ↓
    validation
       ↓
      outcome

Each branch can produce perfectly accurate events. Stored individually, the data could look something like this:

10:00:07  search    success
10:00:08  search    success
10:00:09  tool_call started
10:00:15  tool_call failed
10:00:18  tool_call started
10:00:24  tool_call success
10:00:26  model_call success
10:00:31  validation failed
10:00:37  model_call success
10:00:44  validation success

There is nothing necessarily wrong with those records. They may be exactly what happened. But a flat list does not necessarily preserve why those events exist in relation to one another.

• Was the tool call at 10:00:18 a retry of the failed call, or another branch?
• Did both searches belong to the same fan-out?
• Did the model call at 10:00:37 retry the earlier model call, replace it through a fallback path, or execute because validation created a new stage of work?

Timestamps can help us infer some of this. Operation names, logs, traces and application metadata may provide additional clues. But inference from proximity is different from preserving the relationship itself.

The same set of events can represent different execution structures. A B C D E could have happened as:

A
↓
B
↓
C
↓
D
↓
E

or:

A
├── B
│     └── D
└── C
      └── E

or even:

A
↓
B
↓
C FAILED
↓
C RETRY
↓
E

It's tempting to treat relationships such as parent/child, retry-of or belongs-to-execution as metadata around the real evidence. But for reconstruction, those relationships may themselves be part of the evidence.

A list can tell us what exists. A lineage graph can preserve how those things belonged together. That does not mean every runtime needs to persist an elaborate execution graph. It means that if we expect to answer structural questions later, the structure cannot always be recovered from flat measurements alone.

The design question therefore isn't simply how many events we retain. It's which relationships would become impossible - or dangerously ambiguous - to recover if we didn't preserve them when the execution happened.

And this is where the problem starts to overlap with distributed tracing. A trace already preserves relationships between operations across a distributed system. So if we have tracing, do we actually need another notion of execution lineage at all?

Doesn't Distributed Tracing Already Solve This?

At this point, there is an obvious objection. A distributed trace already exists to connect work across services. If a request moves from an API to a worker, then to a model provider and an external tool, trace context can be propagated across those boundaries so that related operations remain observable as part of a distributed flow. That's exactly what tracing is good at, and I don't think it makes sense to invent a parallel model for information that tracing already preserves well.

A simplified trace might give us something like:

trace
  │
  ├── API request
  │     └── enqueue job
  │           └── worker
  │                 ├── search
  │                 ├── model call
  │                 └── tool call

That is already much richer than a flat list of events. We can inspect timing and reconstruct important technical relationships between operations. With correctly propagated context, those relationships can survive process and service boundaries too.

But there is a subtle distinction between reconstructing technical relationships and identifying the logical unit of work our domain cares about.

Consider a workflow that pauses after the initial trace and resumes later because an external system sends a callback:

trace_01
  │
  ├── request
  ├── planning
  └── external tool request

              time passes

trace_02
  │
  ├── callback received
  ├── workflow resumed
  ├── validation
  └── outcome

Depending on how the system is instrumented, representing those activities as separate traces may be completely reasonable. From the perspective of our domain, however, they may still be two parts of the same logical execution.

The opposite shape is possible too. One technical operation may process a batch containing work associated with several logical executions. Fan-out, messaging and asynchronous processing can create relationships that are not always represented cleanly by assuming one trace is equivalent to one domain execution.

OpenTelemetry accounts for some non-tree relationships through span links. A span can link to other span contexts without making them its parent, which is useful in cases such as asynchronous processing, batching and scatter/gather patterns. That reinforces the point rather than weakening it: distributed execution does not always fit into one simple request-shaped hierarchy.

So I would be careful with an assumption like:

trace_id = execution_id

Sometimes that mapping may be useful. It is not a semantic guarantee we get from tracing itself. A trace answers an observability question about technical operations and their relationships. The application may still have a domain question about which logical execution those operations participated in.

And neither one automatically answers the economic question. Suppose our trace shows that a failed tool call was followed by a retry and that both occurred before the final report was delivered. We have learned something important about the technical execution. We still haven't decided whether the cost of both attempts should be attributed to that report, whether some of the work was shared with another outcome, or which economic boundary the business wants to analyze.

That leaves us with three related but different models:

1. TECHNICAL CAUSALITY - What happened across the distributed system?
2. DOMAIN IDENTITY / LINEAGE - What logical execution did those operations participate in?
3. ECONOMIC ATTRIBUTION - How should the cost of that work be assigned to outcomes, customers, or contracts?