What n8n Execution Traces Can and Can't Tell You About Workflow Coverage

For my own reference, I was trying to answer a simple sounding question about an n8n workflow which branches did this execution exercise? It seems like a lookup given an execution, find the nodes, and for their outputs, observe which items were present. That gives you the exercised branches. This might be possible for some nodes. Then you get to filters. Then you get to loops. Then you realize no, the trace doesn't record everything. It records specific things, elides others, and sometimes gives you enough information to reconstruct what it didn't record. That distinction matters if you're trying to build anything on top of this. The useful question to ask isn't simply what does the trace say? but also what can I prove from it, and what am I assuming? And here's what I found. The obvious setup branch coverage sounds like a lookup Consider this node ┌── true ──> Send email Trigger → IF ────┤ └── false ─> Create task After executing, I'd like to know whether both branches were exercised. A coverage report might then look like this IF ├── true ✓ exercised └── false ✗ not exercised That's useful when testing a workflow, since the UI can tell me that both branches exist, but only an actual execution can tell me whether either of them was exercised. The same applies to a Switch Trigger v Switch / | \ A B C If a test run only uses A, I'd like the coverage tool to be able to report that B and C weren't exercised. That sounds simple enough. The execution trace already has the node outputs, so why not simply read which outputs had items? That's actually the case for branching nodes. The problems begin when you start trying to apply the same logic to every node. So let's talk about what the trace does record. What the trace does record One of the neat things about the execution trace is how node outputs are structured. They're arrays of items, with each item having a JSON field, and multiple outputs per node json {"Switch": [[{ "json": { "id": 101 } }],[],[{ "json": { "id": 103 } }]} Let's say these are the outputs of a switch. Output 0 had an item, output 1 had nothing, and output 2 had an item. That gives us three states output had items the branch was taken output was empty the branch was skipped the node wasn't present at all the branch didn't exist The third state arises from the simplest of causes if the node wasn't executed, it won't appear in the execution trace. So, for branching nodes, it's actually possible to reconstruct a basic coverage report directly from the trace. If IF was executed, and its true output has items, then the true branch was exercised. If the false output is empty, then that branch wasn't taken. And if the downstream node doesn't appear in the trace at all, it wasn't executed in this run. That's actually incredibly useful, since it means that for IF like nodes, coverage is directly observable. The graph can tell you that a branch exists, but only the execution can tell you whether it was taken. For most other nodes, those are orthogonal pieces of information. That's why I'm tempted to think of this as the simplest possible coverage report. But there's a trap waiting for us here, and it's in the third state absence from the trace. An empty output means something specific, but an absent node means something else entirely, and the trace won't tell you which. That's important enough that we'll come back to it. Let's look at nodes that actually process items and see what the trace can and can't say about them. What the trace doesn't record Let's take a Filter: Input v Filter v Output Let's say that 100 items were input into the filter, and the condition was met for 70 of them. The trace will then have the 70 items in the output. But there is no field in the trace that would have the number 100 in it, so the number of input items is not directly observable. That means the trace can't tell you how many items were dropped by the filter. It can help you reconstruct that number, assuming you know the number of input items: 100 - 70 = 30 If the upstream node emitted 100 items, and the filter only emitted 70, it's reasonable to assume that 30 were dropped. But that's an inference that's not directly supported by the trace. This makes all the difference when it comes to tools that report coverage. I think it's much more useful to see something like this Input: 100 (inferred from upstream) Output: 70 Dropped: 30 (inferred) rather than Filter dropped 30 items The reason for this is simple the second bullet implies that n8n itself knows that 30 items were dropped, when in reality, that number was reconstructed from other values. This is especially useful when it comes to loops, since the obvious source of truth about items in them stops being reliable. The loop trap The standard pattern for looping in n8n looks like this ┌─────────────────────┐ │ │ │ ┌─────────────┐ │ Input ──────────────┼──>│ Loop Over │ │ │ │ Items │ │ │ └──────┬──────┘ │ │ │ │ │ v │ │ Loop body │ │ │ │ │ └──────────┘ │ back to │ input └─────────────────────┘ │ Done ───────────────> next node By default, Loop Over Items nodes feed batches of items back into themselves until the loop is completed, with the Done output being used to signal that the loop has completed. This has an impact on the analysis of loop cardinality, since the number of loop iterations isn't directly recorded. The most obvious approach to loop iterations is to infer them from input items Loop iterations = number of items input to loop If true, then with a batch size of 1, a single item would cause the loop to execute once. However, the input to this loop comes from two places: the initial input to the loop node and the loop body itself. A naive approach to counting loop iterations would then mistakenly count the number of items input to the loop as the number of loop iterations. For example, if you used this pattern to implement a map Loop iterations = number of items input to loop node then a cardinality analysis of the loop body would erroneously report that the loop executed more times than it actually did. The reason for this has to do with how feedback connections work in n8n. If you're trying to count loop iterations, the number you're looking for is not the number of items input to the loop node, but the number of items dispatched from its loop slot Loop iterations = number of items dispatched from loop slot Using the cardinality of the loop body as a proxy for loop iterations is not reliable, since the loop body may have other inputs besides the loop node itself. This is why I think of it as the cardinality of the loop dispatch itself. Conceptually, it's similar to this input v ┌───────────────┐ │ Loop Over │ │ Items │ └───────┬───────┘ v loop body └───────────────┐ v loop input again Done ───────────────> after completion The input to the loop body isn't a new original input item, but a re used item from the loop output. This means that the same cardinality tracking rules that applied to the filter don't automatically apply to loops. You have to examine the context of the node and, specifically, the slot you're looking at, since each slot represents a different kind of connection. That brings us to the next subject what you can't observe. What you can't observe The example that I think has the most interesting edge case is a missing Loop Over Items node. If it's not in the execution trace, does that mean that it was reached with zero items, or that it was never reached at all? There is no flag in the trace that would distinguish between the two cases, which means that a coverage tool can't know for sure which one is the case. This means that a branch coverage tool can't reliably distinguish between - the loop was reached, but no items were present when it was executed and - the loop was never reached. There's no separate "never executed" marker in the trace. So the best coverage report for such a case is this Loop: UNOBSERVABLE this might be frustrating for the user, but it's the correct report given the information available in the trace. The loop could have been executed zero times: either because it had no items to process, or because it was never reached. This is an example of why I think such a distinction is useful for coverage tools, especially the ones that have to run on top of the same execution traces that n8n uses. When you start writing tools that need to reason about arbitrary node executions, you find that binary outcome was it executed isn't always useful. This is especially the case when the tool can't look at the node's code, only at the execution trace. That's where the unobservable outcome is useful. I think this can be especially valuable in testing: a test report that knows when it can't observe certain outcomes is much more useful than one that tries to pretend that it knows everything. Two CLI traps There are also two gotchas when it comes to using the CLI to drive execution in scripts. --rawOutput isn't as raw as the documentation suggests The --rawOutput option is supposed to suppress any extra text and only print the JSON, but that's not always the case. In particular, n8n may print some diagnostic information at the start of the execution. So a script like this n8n execute --rawOutput ... | jq is not guaranteed to parse the JSON correctly, since some diagnostic text may appear above it. This was the case in the 2.33.4 release, which printed a runner identifier line like this Runner ID: abc123 So a naïve Python script that tried to do this: python import json json.loads(stdout) would fail. It's better to process the stream more carefully, for example, by finding the actual JSON in the stream. That said, this approach may not be necessary if you're writing the tool yourself. The point is that the CLI's behavior may change, and if you're writing automation around it, you should be prepared to handle those changes. A good rule of thumb is not to make assumptions about the machine parseable output format. If you're writing a script th