5 Content Lanes, One Watchdog: How I Stopped Wondering If My Automation Still Runs

Every morning I used to open my logs and ask the same question: did it actually run last night? The scripts always reported success. The articles were sometimes 186 bytes of an error message. This is how I replaced that daily anxiety with a single shell script. Why this design works When I first built a content generation pipeline, the first problem I hit was this: I thought it was running, but it had actually stopped. launchd fires the script every morning, but it starts before the network is up and exits silently. The API times out with no response, yet a log file still exists. Claude's token budget runs dry, and the error message flows straight into the output file, leaving the body at zero bytes. All of these failures leave behind nothing but the fact that "the script ran." What does it mean to build an environment rather than do work? My answer was a design principle: prove completion by the existence of a file. Not what the script wrote to the log - only whether ~/.claude/logs/.article-daily-done-20260710 exists is treated as truth. That's the essence of the done-marker pattern. content-watchdog.sh is what happens when you extend that idea across all five content lanes. It gets invoked multiple times a day and does one simple job: check the done-markers for every lane, and restart only the ones that are missing. It doesn't break precisely because it's simple. Automation gets stuck for operational reasons more often than technical ones. If you design on the assumption that "it worked yesterday, so it'll work today," it quietly dies on the morning the Wifi isn't connected, at midnight when the battery is at 3%, at the end of the month when the budget runs out. Having a watchdog freed me from the nagging worry that "it should still be running today." The reason I can keep 10 iOS apps going in parallel and hold ¥1.2M/month in revenue is that the time spent on verification is as close to zero as it gets. The overall flow Here's the relationship between the watchdog and the individual lane scripts. launchd (複数スロット) │ └─→ content-watchdog.sh (sweep モード) │ ├─ acquire_lock() # mkdir 競合ロック │ └─ ~/.claude/locks/content-watchdog.lockd/ │ ├─ for lane in article note maker series ameba │ │ │ ├─ done_lane() # done-marker / .done ファイルを確認 │ │ │ └─ [未完了なら] run_capped 1800 bash apply │ │ │ └─ 各レーンスクリプトが自前のdone-markerを立てる │ 例: ~/.claude/logs/.article-daily-done-20260710 │ └─ [全レーン完了なら] send_heartbeat_once() └─ Discord通知 + ~/.claude/logs/.content-watchdog-heartbeat-20260710 Let's read through the actual code in order. The done-marker check logic The done_lane() function has different check logic for each of the five lanes. done_lane() { local lane="$1" local hit case "$lane" in article) [ -f "$LOG_DIR/.article-daily-done-$TODAY" ] ;; note) hit="$(find "$LOG_DIR/note-daily" -maxdepth 1 -name "$TODAY-.done" -print -quit 2>/dev/null || true)" [ -n "$hit" ] ;; maker) hit="$(find "$LOG_DIR/maker-daily" -maxdepth 1 -name "$TODAY-.done" -print -quit 2>/dev/null || true)" [ -n "$hit" ] ;; series) [ -f "$LOG_DIR/.series-daily-done-$TODAY" ] ;; ameba) local ad td f ad="$HOME_DIR/Desktop/Article/ameba" td="$(date +%F)" grep -rlq "created:.$td" "$ad" 2>/dev/null && return 0 for f in "$ad"/.md; do [ -f "$f" ] || continue [ "$(stat -f %Sm -t %F "$f" 2>/dev/null)" = "$td" ] && return 0 done return 1 ;; esac } article and series are managed with a single hidden file (.article-daily-done-20260710 ). note and maker use a 20260710-*.done pattern, a design that allows multiple files to exist. Only ameba has no done-marker and instead checks the actual artifact directly (the created: metadata in .md files, or the mtime). This is a fallback implementation, needed because the ameba script has no done-marker spec of its own, and it functions as "a realistic compromise for wiring an existing script that lives outside the design into the watchdog." When does the done-marker get set? Looking at the implementation of article-daily-stock.sh , the timing of the done-marker is clear. DONE_MARKER="$HOME/.claude/logs/.article-daily-done-${TODAY}" The path is fixed at the top of the script, and after the article body has passed generation, validation, stock placement, and secret scanning, the marker is set before the git push (line 453). # 生成成功＝この時点で当日doneを確定する。以降のgit pushはbest-effort（失敗しても生成は成功扱い） # なので、git失敗でマーカー未設定→翌スロットで重複生成、という事故を防ぐためここで先に立てる。 touch "$DONE_MARKER" The comment says it's "to prevent the accident of git failure → marker unset → duplicate generation in the next slot." As long as push is best-effort, judging done by push success causes duplicate generation. The design principle that "generation and delivery are independent responsibilities" shows up right here. Contention locking with mkdir Since the watchdog itself can be launched from multiple slots, it uses mkdir for mutual exclusion. acquire_lock() { if mkdir "$LOCKDIR" 2>/dev/null; then printf '%s\n' "$$" > "$LOCKDIR/pid" trap release_lock EXIT INT TERM return 0 fi local now mod age now="$(date +%s)" mod="$(stat -f %m "$LOCKDIR" 2>/dev/null || printf '%s\n' "$now")" age=$((now - mod)) if [ "$age" -ge 1800 ]; then log "lock stale age=${age}s; taking over" rm -rf "$LOCKDIR" if mkdir "$LOCKDIR" 2>/dev/null; then printf '%s\n' "$$" > "$LOCKDIR/pid" trap release_lock EXIT INT TERM return 0 fi fi log "lock held; skip" exit 0 } mkdir is an atomic operation at the POSIX level. Even if two processes call mkdir simultaneously, only one succeeds. No flock needed, no Linux/macOS compatibility issues, and the simplicity of exit 0 immediately on failure is its strength. If the lock has been sitting untouched for 1800 seconds (30 minutes) or more, it judges that "the previous process died abnormally and left the lock behind" and forcibly takes over. The same 1800 seconds is used as the timeout for each lane's script invocation, so the next watchdog won't start unless a healthy watchdog has just finished its run. article-daily-stock.sh also has its own lock using the same approach. LOCKDIR="$HOME/.claude/locks/article-daily.lock" if ! /bin/mkdir "$LOCKDIR" 2>/dev/null; then oldpid=$(cat "$LOCKDIR/pid" 2>/dev/null || true) if [ -n "${oldpid:-}" ] && kill -0 "$oldpid" 2>/dev/null; then log "別インスタンス実行中(pid=$oldpid) - skip"; exit 0 fi rm -rf "$LOCKDIR"; /bin/mkdir "$LOCKDIR" 2>/dev/null || exit 0 fi echo $$ > "$LOCKDIR/pid" trap 'rm -rf "$LOCKDIR"' EXIT INT TERM This one adds a PID liveness check. If the previous process is alive it genuinely skips; if it's dead, it force-releases the lock and runs itself. Because the watchdog's lock and each lane script's lock exist as two separate layers, "the watchdog is trying to restart" and "the previous article generation process is still running" don't interfere with each other. The design of Discord notifications and the heartbeat The design of notifying on failure but only once a day on success is baked into send_heartbeat_once() . send_heartbeat_once() { [ "$MODE" = "sweep" ] || return 0 if [ -f "$HEARTBEAT_MARKER" ]; then log "heartbeat skip already_sent=1" return 0 fi notify alerts "💓 content-watchdog heartbeat: 全レーン当日生成済み" touch "$HEARTBEAT_MARKER" log "heartbeat sent" } The watchdog gets invoked many times a day. If all lanes are healthy, it sends a 💓 to Discord only the first time and short-circuits on the marker afterward. Failure notifications, by contrast, fire every time. if [ -n "$FAILED_LANES" ]; then log "result=unhealthy lanes=$FAILED_LANES" if [ "$MODE" = "sweep" ]; then notify alerts "🚨 content停止: $FAILED_LANES 当日未生成(自己修復不能)" fi If the done-marker still isn't set after attempting auto-repair (restarting inside run_lane , then checking done_lane() once more), it tells Discord that manual intervention is required. Looking at the internals of run_lane , the "restart → recheck" cycle fits into a single function. run_lane() { local lane="$1" local script rc script="$(script_for "$lane")" if done_lane "$lane"; then log "lane=$lane status=healthy action=skip" return 0 fi if [ ! -f "$script" ]; then log "lane=$lane status=missing script=$script" return 1 fi log "lane=$lane status=missing-done action=reinvoke script=$script timeout=1800s" if [ "$lane" = "ameba" ]; then run_capped 1800 bash "$script" >> "$LOG" 2>&1 else run_capped 1800 bash "$script" apply >> "$LOG" 2>&1 fi rc=$? log "lane=$lane reinvoke_exit=$rc" if done_lane "$lane"; then log "lane=$lane status=healthy-after-reinvoke" return 0 fi log "lane=$lane status=failed-after-reinvoke" return 1 } The flow is: check with done_lane → skip if fine → restart if not → check with done_lane again. The exit code of the restart (rc ) is written to the log, but it isn't used for the decision. The strictness of "even with exit code 0, no done-marker means failure" is what catches the case where a script spits an error into the output file and exits normally. Implementation details Verifying the artifact is "an article with actual content" in three layers I explained earlier that run_lane() uses the done-marker as its trust basis. So what gets checked before the done-marker is set? The validation layer in article-daily-stock.sh is the answer. First, article_ok() checks the file's existence and its contents. MIN_ARTICLE_BYTES=1200 article_ok() { local f="$1" [ -s "$f" ] || return 1 [ "$(stat -f%z "$f" 2>/dev/null || echo 0)" -ge "$MIN_ARTICLE_BYTES" ] || return 1 grep -qE '^title:' "$f" || return 1 grep -qiE 'request timed out|不明な商品|TODO: *本文|(生成失敗)' "$f" && return 1 return 0 } Four guards are chained in series: does the file exist, is it at least 1200 bytes, does the frontmatter have a title line, and is it free of timeout wording or generation-failure phrases? Keep that last grep -qiE in mind - it connects directly to a story about getting stuck later. Thumbnail verification is handled by a separate function, thumb_ok() . thumb_ok() { local f="$1" w; [ -s "$f" ] || return 1 w=$(sips -g pixelWidth "$f" 2>/dev/null | awk '/pixelWidth/{print $2}') [ -n "$w" ] && [ "$w" -ge 2000 ] 2>/dev/null } sips is macO