My probe passed because it could not fail

Originally published on hexisteme notes. I run pre-registered checks against a live system, read the verdict, and move on - that's the whole point of pre-registering them, so I don't get to argue with the result after the fact. Most of the time the discipline pays for itself. This time it passed, and the pass was wrong, and the reason it was wrong is more interesting than the failure itself: the check could not have returned anything else, whatever had actually happened to the file under test. The question I was probing something narrow: does a hand-made audio crossfade survive a round trip through DaVinci Resolve? Build a timeline with a crossfade sitting on a cut, export it to FCPXML 1.10, re-import it, and see whether the crossfade is still there. Third-party documentation says transitions are invisible to and unmodifiable by the scripting API. Believing that, I pre-registered a judgment method that never looks at timeline structure at all: render audio around the splice and classify it by waveform shape. The judge, exactly as pre-registered: render two seconds either side of the cut, downsample to 8 kHz mono, compute a 20 ms sliding-window RMS envelope - 202 windows across the render - and take the largest normalized step between adjacent windows. Above 0.5, call it a hard cut: the fade is gone. Below 0.5, call it a gradual ramp: the fade survived. The probe came back pass - gradual ramp, max step 0.4761, under the 0.5 threshold. Exit 0, all green. The crossfade had actually been lost at the export step. The pass was a false confirm, and I only found that out by going back in with a second, read-only inspection after the fact. Why the check could not fail The prep instructions for this probe - which I also wrote - said the easiest way to get two adjacent audio items with enough handle to build a crossfade is to take one continuous clip and blade-split it in the middle. That's a completely reasonable instruction on its own. A crossfade needs overlap media on both sides of the cut, and splitting a continuous take is the cheapest way to guarantee that. It also quietly destroys the judge. If both sides of the boundary come from the same continuous recording, then losing the crossfade doesn't produce a hard edge in the waveform - it just reconnects the same continuous audio it started as. Render across that junction and you get a smooth signal whether the fade survived or not. Gradual ramp either way. The check was going to say pass regardless of the real answer, because the one input feature it needed in order to discriminate - a genuine discontinuity at the boundary - was never present to begin with. The judge's implicit precondition was never written down anywhere in the pre-registration: the content on the two sides of the boundary has to actually differ, or a hard cut has nothing to show up as. Nobody checked that precondition against the prep procedure, because the same person designed both, and designing both feels like it should make them consistent by construction. It doesn't. A metric and the data-generating procedure that feeds it are two separate design decisions, and their interaction needs to be checked on its own. Two flags already sitting in the passing run The part that stings is that this wasn't hidden. Two pieces of evidence that the pass was hollow were already sitting in the same results file the pass came from. First, where the largest step happened. The junction was at 2.00 s into the render. The maximum step was at 0.38 s - nowhere near it. None of the top five steps in the whole window fell within ±0.25 s of the junction. The metric wasn't measuring the cut at all; it was picking up ordinary dynamics in the audio elsewhere in the clip. Second, a keyword scan of the exported XML for anything fade- or transition-related came back with zero hits. Neither of these shows up if you only read the top-level verdict. pass is one bit. The evidence that the bit was meaningless was sitting a few fields deeper in the same file, and it only took opening the observations instead of the status to see it. What actually settled it Once the waveform judge was in question, the real answer came from reading the artifacts directly instead of rendering audio and guessing at their shape from the outside. The exported FCPXML was small - 2,595 bytes - and its content was unambiguous: three asset-clip elements and a gap. No transition element, no fade element, anywhere in the file. Timeline geometry closed the loop. Before export, the original timeline had a one-second crossfade - 24 frames at 24 fps - straddling the cut, spanning frames 89180 to 89204 with the cut itself at frame 89192. After the export/import round trip, the two clips butted together at exactly frame 89192, with no overlap. That's the signature of a transition getting flattened: the overlap collapses down to the cut point it was centered on. The 12-frame difference between the pre-export boundary (89180, where the overlap began) and the post-import boundary (89192, the cut center) is exactly half of the one-second fade - the missing overlap, accounted for. Verdict: the crossfade is lost at the export stage, in FCPXML 1.10, on this Resolve build. I didn't test AAF or OTIO export, and I'm not claiming this generalizes to them. The bonus find Reading geometry turned up something I wasn't looking for. The documentation I'd trusted said transitions are invisible to and unmodifiable by scripts. Once I actually queried the track's item list, the transition was right there - exposed as its own item, with its exact name returned by the API: "크로스 페이드 +3 dB," the Korean-localized UI string for "Cross Fade +3 dB," plus accurate start, end, and duration. Readable. Still not writable - I couldn't create or modify one through the API - but readable, which the documentation I'd been working from doesn't say. That's specific to this build and this transition type; I'm not asserting it as a general claim about the scripting API. There was a trap hiding in that same discovery. An earlier boundary calculation - sort items by position, treat each adjacent pair as a clip boundary - silently miscounted, because it treated the transition as if it were a clip. Of the four items the track actually returned, only three were clips. The transition was the fourth, sorted right in among them. What I'd take to another codebase None of this is specific to video timelines or waveform envelopes. The shape recurs anywhere you write a check against a system you don't fully control: - A test that asserts on a value the test itself produced - the input was generated by the same code path being tested, so of course they agree. - A mock that just echoes back whatever the code under test wrote to it - the mock can't disagree with the code, because it has no independent source of truth. - A golden file regenerated from the current output instead of a trusted baseline - the diff against "golden" is now a diff against a copy of itself. All three share the same defect this probe had: the check's input is entangled with the thing being checked, so there is no reachable state in which the check returns anything but pass. That's not a check. It's a formality wearing a check's clothes. Three habits would have caught this earlier, or at least caught it faster: - Before trusting a check, ask what it would take for it to fail - using the exact input you're about to give it, not a hypothetical one. A judge's discriminative power rests on a precondition, and preconditions left implicit are exactly what the most convenient prep procedure tends to quietly violate. - When the artifact is inspectable, prefer structure over signal. A waveform classifier summarizing a few seconds of audio into one number is strictly less informative than the exported file format and frame geometry, when both are available. I only needed the second because I'd mistakenly trusted the first. - Read the observations, not just the status. Both clues that overturned this pass - the mislocated step, the empty keyword scan - were sitting in the same results file the verdict came from. Nobody had to go looking anywhere new. - When the same person designs the prep procedure and the metric, that interaction is a checklist item, not an assumption. "Can this metric discriminate, given this exact prep?" is a question that has to be asked out loud, because internal consistency between two decisions made by the same person is not something you get for free. One caveat on the fix itself: I haven't re-validated the waveform methodology end to end. What overturned the pass was a one-time structural read - export XML plus frame geometry - not a rebuilt, re-run judge. If the waveform check is going to be trustworthy going forward, it still needs prep that produces genuinely different content on either side of the boundary, and that fix hasn't been exercised yet. FAQ Q. How can a passing test be meaningless? When the test's input can't produce the failure signature no matter what the real state of the system is. Here, both a surviving and a lost crossfade rendered as the identical waveform, because the prep made both sides of the boundary the same underlying audio. The test wasn't wrong about what it measured - it measured a quantity that happened to be uninformative for this question. Q. Wasn't the fix just "read everything manually"? No - the fix is structural, not manual: for artifacts you can parse (an XML export, frame geometry from an API), prefer that over a signal heuristic when both are available. The manual step here was realizing the heuristic needed a second opinion at all, which came from noticing two anomalies already present in the machine-produced results. Q. Does this mean the vendor documentation was flat wrong? Only for the case I tested: this Resolve build, this transition type, read through this specific API call. The transition was readable, not writable, and I'm not extending that past what I actually queried. More not