Nothing new. Exit code 0.

Originally published at tothenoc.substack.com

A check that cannot fail is not a check.

Last week a check told me there was nothing new in my mailbox. There were four messages that mattered, all from that same day. The check did not fail. It ran, it filtered, it returned an empty list.

The filter compared dates written as text, and the text did not match. Empty, no error. I caught it by reading the mailbox myself.

It was not the first time

Same family, different tools, all on our own systems:

• a search over the mail server returned zero results. The mailboxes are stored compressed. The search read the raw files, found nothing, and said so politely.
• an upgrade script exited with code 0, one second after it started. It had checked who was running it, did not like the answer, and quietly did nothing.
• a daily report came out as a file of zero bytes. Its output path pointed to a folder that had been archived the night before.

None of these crashed. A crash is easy, somebody sees it. The dangerous result is the one that looks exactly like good news: nothing found, nothing changed, nothing to report.

What we do now

• every check has a positive control, a known item it must find. If the check cannot find the thing I know is there, its zero means nothing,
• a result that came back too fast is not a result. Zero hits on a full disk in zero seconds is a tool that did not look,
• read the log, not the exit code. Code 0 says the program ended. It does not say what it did,
• a report has a last line that says it finished. No last line, no report.

"Nothing new" is a claim. Before you believe it, ask how the check would look if something was new.

If you are deciding who watches this for you, this is what to ask: what an outsourced NOC actually covers, and what it costs.

-- Alex