Two Bugs, Two Strangers, One Week: What Shipping Early Actually Buys You

A week ago I put a rough, honestly-a-bit-thin version of PulseWatch in front of real people for the first time. Within days, two different strangers - independently, unprompted - found two real gaps in it. Neither was catastrophic. Both were exactly the kind of thing you only find by watching someone else use the thing you built. This is the story of both, and the fixes.

Bug one: the run that never ends

This first bug came from a friend testing it on a real script. His question was simple: "What happens if start fires twice before end?" Good question. At the time: nothing good. Here's why.

PulseWatch works on two pings - a job calls /start when it begins and /success (or /fail) when it's done. The server tracks whichever run is currently "open" for a monitor. The bug: if a job's process restarts mid-run - a crash-and-retry, a redeploy that catches it mid-flight, a scheduler firing twice - you get a second /start before the first run ever closes. The old run just sits there, open forever, an orphan with no ending. Worse, because the watchdog was still waiting on that run's expected finish time, it could fire a false "still running" alert for a run that was, for all practical purposes, dead and abandoned.

The fix is a small rule with an outsized effect: a new /start supersedes whatever run is currently open. The old run gets marked superseded - a terminal, non-alerting status - and a fresh run begins clean. The watchdog was updated to treat superseded as a dead end: nothing to wait on, nothing to alert about, and it never shows up in a user's run history. It's not a failure and it's not a success. It's just "this run doesn't matter anymore, a newer one replaced it."

The logic, roughly:

def handle_start(monitor):
    open_run = monitor.get_open_run()
    if open_run is not None:
        open_run.status = "superseded"
        open_run.finished_at = now()
    new_run = Run(monitor=monitor, status="running", started_at=now())
    db.session.add(new_run)
    db.session.commit()

Simple once you see it. Invisible until someone actually restarts a job mid-flight and asks the right question.

Bug two: the failure that isn't a failure, exactly

The second came from a comment on my first post here on Dev.to, from someone whose day job is infrastructure monitoring. He'd read the piece on catching jobs that die silently and asked the harder follow-up: "How does PulseWatch deal with jobs that aren't dead, just flaky, before alert fatigue kicks in?"

That's a different failure mode entirely, and at the time I only had half an answer. PulseWatch already alerts on state transitions, not conditions - one email when a monitor goes down, one when it recovers, not a repeat every check cycle. That solves the "job has been down for six hours, stop emailing me about it" problem.

What it didn't solve: a job that's actually unstable - fails, recovers, fails, recovers, each transition genuinely real. Alert on every transition, and you get exactly what it sounds like: fail, recover, fail, recover, five emails in twenty minutes, and you mute the whole channel by Thursday. Technically correct alerting, practically useless.

The naive fix - just suppress alerts if a monitor's been unstable recently - has an obvious failure mode of its own: it delays the one alert you actually wanted immediately, buried under a threshold meant for noise. That tradeoff is exactly what I flagged as unsolved in my reply to him at the time.

Here's where I landed. Rather than alerting on every up/down transition, PulseWatch now watches a rolling window and counts transitions within it. Cross a threshold - four or more state changes within two hours, by default - and the monitor enters a distinct flapping state. Two things change once it's flapping:

• One alert fires on entry - "this monitor is unstable," not "this monitor is down," a meaningfully different signal.
• Ordinary up/down alerts are suppressed while flapping continues, and one more alert fires when it settles back below the flap threshold.

So a genuinely flaky job gets exactly two emails - "it's flapping" and "it's settled" - no matter how many times it actually flipped in between. A job that's just plain down still gets its immediate down alert, because that's a single transition, not a pattern - the threshold only kicks in once instability itself becomes the story.

The state machine, simplified:

FLAP_WINDOW_MINUTES = 120  # rolling window to look back over
FLAP_THRESHOLD = 4         # this many state changes in the window = flapping

def is_currently_flapping(monitor):
    """Count how many times status changed across recent runs.
    Flapping if it crosses the threshold within the window."""
    runs = monitor.recent_terminal_runs(minutes=FLAP_WINDOW_MINUTES)
    # oldest → newest
    changes = sum(1 for prev, cur in zip(runs, runs[1:]) if prev.status != cur.status)
    return changes >= FLAP_THRESHOLD

def check_monitor(monitor):
    currently = is_currently_flapping(monitor)
    if currently and not monitor.is_flapping:
        monitor.is_flapping = True
        alert(monitor, "unstable - flapping between states")
    elif not currently and monitor.is_flapping:
        monitor.is_flapping = False
        alert(monitor, "stabilised")
    elif not monitor.is_flapping:
        # normal single-transition alerting, unchanged
        handle_normal_transition(monitor)
    db.session.commit()

The one new piece of state this needed - a boolean is_flapping column - became, almost by accident, the first schema change to go through a proper migration pipeline I'd just set up on the live database. Small feature, useful excuse to prove the plumbing works before something bigger needs it.

The actual point

Neither of these bugs was findable by staring at the code longer. The double-start case only shows up when a real process actually restarts mid-run. The flapping case only matters once you've watched enough alert emails pile up to feel the fatigue yourself. Both needed someone else's script, someone else's failure pattern, someone else's patience to ask "but what about-".

That's the actual case for shipping something rough and putting it in front of people early, not the version of the advice that's become a cliché. It's not just "get feedback." It's that some categories of bug are structurally invisible to the person who wrote the code, because the code matches their own mental model of how it'll be used - and the whole value of a stranger is that their mental model is different.

Both fixes are live now. If you're running unattended jobs - cron, scheduled scripts, agent pipelines - and any of this sounds familiar, PulseWatch is free to try, and I'd genuinely like to know what breaks next.