Empty Is Not a State

Your poller returned None . What happened? Maybe the source published nothing new. Maybe the server went down. Maybe the endpoint returned HTML instead of RSS and the parser quietly produced zero items. Maybe the credentials expired. Maybe the API retired the endpoint. Or maybe you do not have enough evidence to know yet. These are different facts. They require different retry policies, different alerts, and different state transitions. Yet many production systems collapse all of them into one value: result = fetch(source) if result is None: source.done = True This is not mainly a logging problem. It is a control-flow problem. The moment None is allowed to drive a state transition, the system destroys the distinction between absence, failure, and uncertainty. A temporary outage can permanently stop monitoring. A parser regression can look like a quiet feed. An expired credential can trigger endless retries against an endpoint that is perfectly healthy. The fix is not complicated. Record what happened, interpret it separately, and only mutate durable state when the evidence supports the transition. Observation. Adjudication. Verdict. One empty result, at least five different facts Consider a scheduled job polling an external feed. The first possibility is genuinely boring: the request succeeded, the parser succeeded, and the source contains no item newer than the last one you processed. The correct action is to update the last successful check and poll again normally. The second is temporary unavailability. DNS failed, the connection was refused, or the server returned 503 . That needs backoff and eventually an alert. It is not evidence that the source is finished. The third is a contract failure. The server returned 200 and some bytes, but the parser could no longer extract what it previously extracted. Perhaps the XML changed. Perhaps a CDN returned an HTML challenge page. Perhaps the API schema moved. Retrying the same parser every fifteen minutes will not repair it. The fourth is an access or capacity limit. 401 , 403 , and 429 do not describe the content. They describe your ability to retrieve it. The response may require new credentials, a slower request rate, or a quota decision. The fifth is the one production code tends to resist: you do not know. The response is ambiguous, the source has no history, or the available signals contradict each other. Undetermined is not an implementation failure. It is a legitimate state in the model. All five can arrive at the caller as None . That is the design smell. The fetcher should not decide what the fetch means The boundary component should record what it observed without trying to convert it into business meaning. from dataclasses import dataclass from datetime import datetime @dataclass(frozen=True) class Observation: source_id: str observed_at: datetime http_status: int | None transport_error: str | None content_type: str | None bytes_received: int items_parsed: int | None parse_error: str | None newest_item_date: datetime | None etag: str | None Every attempt produces an Observation , including the failed ones. A timeout is data. A 200 response containing 48 KB of HTML is data. A parser that never ran is different from a parser that ran successfully and found zero items. This distinction is easy to erase with exception-driven code: try: return parse(fetch(url)) except Exception: return None That function has converted several independent failure domains into the same value. By the time the scheduler receives it, the useful information is already gone. Empty is only readable against history A single observation is often insufficient. A 200 response with zero parsed items can be normal for one source and a strong regression signal for another. You need a baseline. @dataclass(frozen=True) class Baseline: last_success_at: datetime newest_item_date: datetime | None typical_item_count: int | None expected_content_type: str | None etag: str | None = None Suppose a feed has returned between 15 and 25 items on every poll for six months. Today it returns 200 , 42 KB of data, text/html , and zero parsed items. Calling that “no new content” would be an impressive act of optimism. Now take the same response from a source you have never seen before. It may be broken, or it may simply not be a feed. You cannot infer a contract regression because you have no known contract to compare against. Same response. Different evidence. Different verdict. This is why history is not just useful metadata. It changes what can be concluded from the current observation. A verdict is not another boolean The adjudicator combines the observation with the baseline and produces a result the scheduler can act on. from enum import StrEnum class Outcome(StrEnum): ITEMS_FOUND = "items_found" NO_NEW_ITEMS = "no_new_items" UNAVAILABLE = "unavailable" CONTRACT_FAILURE = "contract_failure" ACCESS_LIMITED = "access_limited" ENDPOINT_GONE = "endpoint_gone" UNDETERMINED = "undetermined" @dataclass(frozen=True) class Verdict: outcome: Outcome observation: Observation baseline: Baseline | None reasons: tuple[str, ...] @property def is_settled(self) -> bool: return self.outcome in { Outcome.ITEMS_FOUND, Outcome.NO_NEW_ITEMS, } I deliberately did not put confidence: 0.83 in this version. Unless those scores are calibrated against labelled outcomes, confidence is usually just intuition wearing a decimal point. A deterministic classifier can still express uncertainty without inventing measurement. It can return UNDETERMINED , preserve its reasons, and wait for more evidence. If you have historical data and can demonstrate that verdicts emitted at 0.8 are correct roughly 80% of the time, add calibrated confidence. Until then, evidence is more useful than decorative precision. The adjudicator The rules do not need to be clever. They need to preserve distinctions that affect downstream behaviour. def adjudicate( obs: Observation, baseline: Baseline | None, ) -> Verdict: def verdict(outcome: Outcome, *reasons: str) -> Verdict: return Verdict(outcome, obs, baseline, reasons) if obs.transport_error: return verdict( Outcome.UNAVAILABLE, f"transport error: {obs.transport_error}", ) if obs.http_status in {401, 403, 429}: return verdict( Outcome.ACCESS_LIMITED, f"HTTP {obs.http_status}", ) if obs.http_status in {404, 410}: return verdict( Outcome.ENDPOINT_GONE, f"HTTP {obs.http_status}", ) if obs.http_status is not None and obs.http_status >= 500: return verdict( Outcome.UNAVAILABLE, f"HTTP {obs.http_status}", ) if obs.http_status == 304: return verdict( Outcome.NO_NEW_ITEMS, "server confirmed that the representation was not modified", ) if obs.http_status != 200: return verdict( Outcome.UNDETERMINED, f"unhandled HTTP status: {obs.http_status}", ) if obs.parse_error: return verdict( Outcome.CONTRACT_FAILURE, f"parser failed: {obs.parse_error}", ) if obs.items_parsed == 0: if baseline is None: return verdict( Outcome.UNDETERMINED, "zero items with no historical baseline", ) if baseline.typical_item_count and baseline.typical_item_count > 0: return verdict( Outcome.CONTRACT_FAILURE, "source historically contained items", "current parser returned zero", ) return verdict( Outcome.NO_NEW_ITEMS, "zero items is consistent with the baseline", ) if obs.items_parsed and baseline is None: return verdict( Outcome.ITEMS_FOUND, "first successful observation", ) if ( obs.newest_item_date and baseline and baseline.newest_item_date and obs.newest_item_date str: return { Outcome.ITEMS_FOUND: "ingest and advance the baseline", Outcome.NO_NEW_ITEMS: "record success and poll normally", Outcome.UNAVAILABLE: "retry with exponential backoff; alert after a threshold", Outcome.CONTRACT_FAILURE: "quarantine the result; alert the parser owner", Outcome.ACCESS_LIMITED: "apply rate-limit or credential recovery policy", Outcome.ENDPOINT_GONE: "confirm across repeated observations; then disable", Outcome.UNDETERMINED: "preserve state and collect another observation", }[verdict.outcome] Notice that even ENDPOINT_GONE does not immediately delete or permanently complete the source. A single 404 can come from a bad deployment, a routing error, or an eventually consistent configuration change. The verdict is strong enough to change behaviour, but destructive state transitions should still require repeated evidence or human confirmation. This is the difference between classification and governance. The classifier says what the current evidence most strongly supports. The policy decides what the system is authorised to do about it. Do not let uncertainty rewrite history The baseline influences future decisions, so updating it is not an innocent write. A bad update changes how later observations will be interpreted. Only settled content outcomes should advance it. def advance( baseline: Baseline | None, verdict: Verdict, ) -> Baseline | None: if not verdict.is_settled: return baseline obs = verdict.observation return Baseline( last_success_at=obs.observed_at, newest_item_date=( obs.newest_item_date or (baseline.newest_item_date if baseline else None) ), typical_item_count=( obs.items_parsed if obs.items_parsed and obs.items_parsed > 0 else (baseline.typical_item_count if baseline else None) ), expected_content_type=( obs.content_type or (baseline.expected_content_type if baseline else None) ), etag=obs.etag or (baseline.etag if baseline else None), ) An unavailable source does not become the new normal. A parser failure does not reset the typical item count to zero. An ambiguous first poll does not establish a baseline that will make the second ambiguous poll look valid. That last failure is particularly unpleasant. Once an uncertain observation is written as truth, later logic uses the corrupted history as evidence. The system does not merely remain wrong. It manufactures confirmation for its own mistake. What the cases look like These fixture cases exercise the decisions that used to share one return value: | Case | Observation | History | Verdict | Action