When the exit code lies: fork retractions, lost transactions, and trusting the chain in midnight-node

Your transaction tool reports FAILED_TO_FINALIZE. Your CI marks the job red. Your retry logic fires. But the block explorer says the transaction finalized 22 seconds after you sent it. The exit code lied.

This is a debugging story from midnight-node - Midnight's Substrate-based node - about how a transaction watcher loses track of a transaction during a chain fork, why the failure is invisible in tests, and the pattern that fixes it: on timeout, ask the chain itself, not the stream that was supposed to tell you about it. The fix landed as PR #1927 for issue #1854.

Forks are normal, your tooling forgets

Midnight produces blocks with AURA every 6 seconds and finalizes them with GRANDPA a couple of blocks later. Between production and finality, the chain is allowed to disagree with itself: two validators can build competing blocks, and one branch eventually wins. A transaction included in the losing branch is retracted - kicked back to the transaction pool - and normally re-included in a block on the winning branch a few seconds later.

From the chain’s perspective this is routine housekeeping. From your tooling’s perspective, it’s a trap.

The toolkit’s send command watches a submitted transaction with subxt’s watch stream, which emits status events: InBestBlock, NoLongerInBestBlock, InFinalizedBlock, and so on. The sender’s logic was:

• Wait for the tx to appear in a best block.
• Then wait (with a timeout) for InFinalizedBlock.
• Timeout expires → report FAILED_TO_FINALIZE, exit non-zero.

Failure mode

Here’s the failure mode from the issue, reconstructed from logs:

t+0s   tx submitted, InBestBlock (block A)
t+6s   fork: block A retracted → NoLongerInBestBlock
t+8s   tx re-included in block B on the winning branch
t+22s  block B finalized  ✅ tx is permanently on chain
t+60s  watcher's finalization timeout expires → exit: FAILED_TO_FINALIZE ❌

Two things went wrong. First, NoLongerInBestBlock was being silently swallowed - the logs showed nothing at the moment the interesting thing happened. Second, after the retraction, the watch stream never surfaced the re-inclusion; the watcher waited out its timeout on a dead branch while the transaction quietly finalized elsewhere.

Why a wrong exit code is worse than a crash

If this tool only fed dashboards, a spurious failure would cost an engineer an eyebrow raise. But callers make decisions based on exit codes. The sharpest edge in our case: reward-claim transactions with at-most-once semantics. A wrapper script sees the non-zero exit, assumes the claim never landed, and retries - resubmitting an operation that already succeeded. The exit code isn’t diagnostics; it’s an API, and it was returning wrong answers.

This is a general lesson worth internalizing: any tool that reports transaction outcomes is part of someone’s correctness argument. Treat its outputs with the same rigor as consensus code.

The fix: the chain gets the last word

The watch stream is a convenience, not a source of truth. The source of truth is the finalized chain, and it’s sitting right there behind an RPC. So on watch timeout, the sender now scans finalized blocks, newest first, looking for the extrinsic hash:

pub async fn find_in_finalized_chain(
    client: &MidnightNodeClient,
    extrinsic_hash_hex: &str,
    max_depth: u32,
) -> Option<String> {
    let mut hash = client.rpc.chain_get_finalized_head().await.ok()?;
    for _ in 0..max_depth {
        let block = match client.rpc.chain_get_block(Some(hash)).await {
            Ok(Some(b)) => b,
            _ => return None,
        };
        for ext in &block.block.extrinsics {
            let ext_hash = format!(
                "0x{}",
                hex::encode(sp_crypto_hashing::blake2_256(&ext.0))
            );
            if ext_hash == extrinsic_hash_hex {
                return Some(hash_to_str(hash));
            }
        }
        if block.block.header.number == 0 {
            return None;
        }
        hash = block.block.header.parent_hash;
    }
    None
}

A few design notes:

• Bounded depth (64 blocks below the finalized head - comfortably past any plausible retraction-to-finalization window at 6-second blocks). An unbounded walk to genesis on a wrong hash would turn one bug into a different one.
• Extrinsic identity is just a hash. A Substrate extrinsic’s hash is the blake2-256 of its encoded bytes, so the scan needs no indexer, no storage queries - fetch blocks, hash extrinsics, compare.
• Fail open, loudly. RPC errors during the scan log a warning and return None - the sender then reports the timeout as before. The fallback can only upgrade a false failure into a truthful success, never mask a real one.

With the scan in place, the outcome reporting becomes honest:

let message = if finalized_block_hash.is_some() {
    if finalized.is_some() {
        "FINALIZED"
    } else {
        "FINALIZED_AFTER_RETRACTION"
    }
} else {
    "FAILED_TO_FINALIZE"
};

FINALIZED_AFTER_RETRACTION exits zero - because the transaction is on chain - but it’s a distinct message, so operators can see how often forks are eating their watch streams. And the retraction itself is no longer swallowed: the moment NoLongerInBestBlock arrives, the sender logs tx retracted from best block; watching for re-inclusion.

The timeouts also became configurable (MN_SEND_BEST_BLOCK_TIMEOUT / MN_SEND_FINALIZED_TIMEOUT, in seconds) for slow or fault-injected environments - as environment variables rather than CLI flags, partly to keep the change surface away from other in-flight CLI work.

Testing what you can, admitting what you can’t

Here’s the honest part: you cannot deterministically force a fork retraction in CI. Retractions emerge from validator timing races; no RPC call produces one on demand. A test suite that claims to cover the retraction path end-to-end would be lying the same way the exit code was.

What you can do is split the fix so the untestable part is trivially small (a timeout branch calling one function) and the testable part carries the logic. The e2e test for the scan uses a trick worth stealing: every Substrate block already contains a transaction you didn’t send - the timestamp inherent that the block author injects. So the test:

• runs against a real node, grabs a finalized block, and takes its timestamp-inherent extrinsic;
• hashes those bytes and asserts find_in_finalized_chain locates that extrinsic at exactly that block (positive case, using an extrinsic the test never had to submit);
• asserts a fabricated hash comes back None after the bounded walk (negative case).

No fault injection, no mocked RPC - the scan is exercised against genuinely finalized blocks, and the only unverified wiring is a five-line match.

Takeaways

• Watch streams are best-effort; finality is a fact. When a stream and the chain can disagree, reconcile against the chain before reporting failure.
• Exit codes are API. If a caller might retry based on your answer, a false negative is a correctness bug, not a cosmetic one.
• Never swallow the interesting event. The silent NoLongerInBestBlock was the difference between a 5-minute diagnosis and a mystery.
• Bound your fallbacks. A recovery path with no depth limit is a new incident waiting to happen.
• Be honest about test coverage. Shrink the untestable wiring instead of pretending the test forces the failure.

The fix is open for review at midnightntwrk/midnight-node#1927 - feedback welcome. If you’re building on Midnight and your tooling watches transactions, go check what your code does with NoLongerInBestBlock. There’s a decent chance the answer is “nothing,” and now you know why that matters.