Three ETL failure patterns I now write into the output file, not just the logs

My Reddit scraper returned empty arrays for 71 days. Nothing broke visibly. The market-listening output file kept being written, kept being committed to git, kept looking like a healthy daily snapshot. Nobody caught it until the interpretation layer noticed that Reddit-sourced signals hadn't changed in two months. The three patterns I added afterward are small. Combined, they mean that failure mode can't happen silently again - the artifact itself reports what went wrong, not just the runner logs that nobody reads unless something is already on fire. All three are implemented in scripts/market-listening/collect.mjs, with the 71-day blind spot documented in the inline comments. Pattern 1: sources_ok lives in the artifact, not the logs The daily output JSON now contains a sources_ok object with a boolean per source: { "sources_ok": { "youtube": true, "autocomplete": true, "bluesky": true }, "errors": [], ... } If any YouTube request fails, sources_ok.youtube flips to false . The file that gets committed to git carries its own health signal. That's the core change: the failure state lives in the artifact, not in a log stream. Why this matters over logging: the committed artifact shows up in git diffs, gets reviewed in the daily health check, and is read by the interpretation layer before it uses any of the data. Logs from a GitHub Actions cron are reviewed only if someone suspects a problem. The artifact is reviewed on every downstream read. The pipeline health monitor checks sources_ok on every run and opens a GitHub issue if any source is false . The interpretation layer that reads these files is fail-closed on sources_ok : if sources_ok.youtube === false , the YouTube signals for that day are ignored rather than mixed into the weekly aggregate. Separating collection from interpretation makes this guard practical - the interpretation step reads the artifact and checks health before consuming any of the results. Pattern 2: errors[] alongside the results The boolean sources_ok tells you something failed. The errors array tells you what and when: function recordError(source, detail) { const entry = { source, detail, at: new Date().toISOString() }; errors.push(entry); console.error(ERROR [${source}] ${detail}); return entry; } Every failed fetch calls recordError with the source name, the error detail (including HTTP status codes), and an ISO timestamp. These accumulate in the errors array and are written into the same artifact as the successful results. The practical difference: if a run produces errors: [{ "source": "bluesky", "detail": "HTTP 403", "at": "2026-08-13T07:..." }] , I can see the exact failure mode without opening any runner log. After eight days of that, the error messages have been consistent enough that I know exactly what the Bluesky API is rejecting - not just "bluesky failed." The at timestamp matters too. If a script partially completes and then errors, the timestamps in errors[] tell you at what point in the run things started going wrong, without reconstructing it from log correlation. This is different from the approach I described in catching silent failures in a GitHub Actions pipeline, which focuses on job-level failures in CI. These failures are internal to a single job that mostly succeeds - partial failures within a healthy-looking run, which CI-level checks won't catch. Pattern 3: the vacuous-true guard on .every() The sources_ok flag for YouTube is computed by checking whether every query produced valid voted results: const ok = QUERIES.length > 0 && QUERIES.every((q) => queries[q].reps_ok >= MIN_APPEARANCES && queries[q].accepted > 0); if (!ok) recordError("youtube", "one or more queries produced no voted results"); Without QUERIES.length > 0 , an accidentally-empty QUERIES array would satisfy .every() vacuously - [].every(fn) returns true for any fn . The script would commit an artifact with sources_ok.youtube = true and zero actual data inside. This is more plausible than it sounds. Any config change that blanks the query list - a bad merge, an over-aggressive deduplication, an env substitution that resolves to empty string - would silently produce an artifact that reports itself as healthy. The length guard is three tokens. The same pattern applies to any validation using .every() , .all() , or equivalent over a collection that can be empty. If an empty input is a failure condition - and it usually is - guard on length first. The validation chain length > 0 && every(condition) handles it cleanly. What the three patterns give you together A single fetch failure: sources_ok flips to false. The git diff shows it. The health check catches it. The interpretation layer ignores that source for that day. The error detail explains what happened. A config accident that empties the query list: the length guard trips, the source reports false, the artifact contains no data but reports that explicitly rather than vacuously reporting success. An extended outage (like the current Bluesky block): eight consecutive artifacts, each with sources_ok.bluesky = false and matching error entries. The pattern in the committed history is unmistakable and requires no log archaeology. The 71-day Reddit blind spot was eventually caught from the outside - someone noticed the data wasn't changing. These patterns are designed to make that kind of failure visible from the inside, in the artifact itself, immediately on the day it happens. Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted. Top comments (0)