Cron Dead Man's Switch: My Watchdog Checked Last Week's File

At 07:00 KST on Sunday, August 2, the weekly planning job behind this blog wrote three lines to its log and stopped:

[2026-08-02 07:00] weekly-review start
You've hit your weekly limit · resets Aug 3 at 6pm (Asia/Seoul)
[2026-08-02 07:00] weekly-review finished (exit 1)

No alert fired. Over the next four days this site published 24 posts, every one of them chosen without the plan that job was supposed to write. I noticed on day five — not because anything paged me, but because the annotation no weekly plan file had piled up 23 times in the publishing log. This is the post-mortem, and the argument it left me with: every scheduled job needs a dead man's switch — a monitor that alarms on absence, not on error.

Death by budget, at the worst-scheduled minute

The direct cause is boring. The planner is a headless LLM run, and the account had hit its weekly usage limit, the same ceiling that took out an afternoon of unattended publishing slots in July. The limit reset 35 hours later, on Monday at 18:00. The planner is scheduled for Sunday 07:00 — the tail end of the usage week, after six days of publishing had already eaten the budget. The job that steers everything else runs at the minute the budget is most likely to be zero. That is a scheduling bug in its own right.

But a job dying is normal; the system is built to tolerate it. The interesting failure is the layer underneath.

The watchdog checked last week's file

I knew a headless planner could die quietly, because a launchd job had already failed silently on this machine in July. So the runner script verifies the artifact after the run: the plan lands at plans/<monday>.md, and if that file is missing, the script sends a Telegram alert. Trimmed to the relevant lines:

MONDAY=$(python3 - <<'EOF'
import datetime, zoneinfo
kst = datetime.datetime.now(zoneinfo.ZoneInfo("Asia/Seoul")).date()
print((kst - datetime.timedelta(days=kst.weekday())).isoformat())
EOF
)
PLAN="projects/blog-en/plans/${MONDAY}.md"
if [ ! -f "$PLAN" ]; then
  # Telegram alert: weekly review produced no plan file
fi

The comment above that block in the real script says, translated from Korean: even exit 0 without the artifact is a failure — do not pass over it silently. I wrote that guard specifically against silent failure. It failed silently.

The bug is the textbook idiom for the Monday of the current week. Python's date.weekday() anchors the week on Monday: Monday is 0, Sunday is 6. Run on any working day, d - timedelta(days=d.weekday()) returns the Monday you mean. Run on Sunday, it returns the Monday six days behind you:

>>> d = datetime.date(2026, 8, 2)   # the Sunday the job died
>>> d.weekday()
6
>>> d - datetime.timedelta(days=d.weekday())
datetime.date(2026, 7, 27)

So the guard looked for plans/2026-07-27.md: last week's plan, sitting exactly where last week had left it. Found it. Concluded the run had succeeded. The file the dead run was supposed to produce, plans/2026-08-03.md, was never checked by anything; exit code 1 sat in a log file and the alert branch was never reached.

Three components, one shared assumption

The same Monday arithmetic appears in three places: the gate in the planner's prompt (skip if this week's plan already exists, the idempotency guard every LLM cron job here starts with), the consumer script that publishing slots call to pull the next planned item, and the shell guard above. On a Sunday, all three point backward. Even with budget to burn, the run would likely have seen last week's file as this week's plan and exited without writing anything. I can't prove that branch — the run died before reaching its gate — but the two deterministic components are code, and both provably resolve to the wrong Monday every Sunday.

That is the general lesson. A success check written inside the producer shares the producer's assumptions — my guard reused the very date expression whose semantics were broken, so the job and its watchdog were wrong together. Correlated failure is the one kind a self-check structurally cannot catch.

The fallback worked, which made it worse

Nothing stopped publishing. By design, a slot that finds no plan file falls back to discovering its own topic; the runner's own comment says a missing plan must not halt the pipeline. That fallback engaged in every slot from August 3 through 6, and the output looked normal from outside: 5, 10, 6 and 3 posts a day, deploys green, the nightly revenue report mailing its usual numbers. Each slot did the locally correct thing — called the plan script, got exit code 1, wrote no weekly plan file, self-discovered topic into the log, and moved on.

Twenty-three annotations. Zero alerts. Graceful degradation without absence detection hides an outage indefinitely: the better the fallback, the longer the outage stays invisible. The same week made the point twice, when an expired credential blacked out this fleet for 17 hours and detection came from a neighboring system's symptoms rather than any alarm here. Both incidents have the same shape: alerting fires when an action fails loudly, and nothing fires when an expected event simply does not happen.

What actually detects absence

Two things do, and neither lives inside the producer.

The consumer. The Monday 07:30 publishing slot knows precisely which file it needs and when it should exist. Today it treats a missing plan as routine: fall back, annotate, continue. Correct for one slot, wrong as a steady state. The fix is one conditional — on the first slot of a new week, a missing plan file is not routine, it is the planner's obituary, and the alert belongs there. The consumer's definition of success is independent of the producer's, which is exactly the property my shell guard lacked.

An external dead man's switch. Instead of asking whether the job reported failure, invert the contract: the job must actively report success, and a third party alarms when the report fails to arrive. This is heartbeat monitoring, and Healthchecks.io's cron documentation states the contract in one sentence: the job pings a URL every time it completes, and when the service does not receive the ping at the expected time, it notifies you, with a grace window for jobs that run late. The docs list what this catches that log-based alerting cannot: machine down, cron daemon not running or misconfigured, non-zero exit, task hanging. A missing ping covers my case too — provided the ping sits after a trivial artifact check (does the file exist), not after a re-derivation of the job's own logic.

The quiet failure is the one practitioners fear. In a July 2026 Ask HN thread on monitoring cron jobs, the poster's observation was that most teams have no monitoring on scheduled work at all, and that uptime monitors cannot tell you if your scheduler stopped firing. A 2021 thread on the same question contains my favorite trap, from a user of one of these services: it works, I think — no cron job has failed yet. A monitor you have never seen fire is unverified by definition. Comment out the ping once, wait past the grace period, and watch the alert arrive; that ten-minute drill is the only proof the switch is wired to anything.

Repairs, queued and not applied

Three items went into the repair queue. As of this post going live, none is applied:

1. Fix the date anchor. A Sunday run plans the week that starts tomorrow: d + timedelta(days=(7 - d.weekday()) % 7), which maps Sunday to tomorrow and Monday to itself. All three call sites move to one shared function, because three copies of one expression is how this happened.
2. Alert from the consumer. First slot of a new week, plan file missing: send the Telegram message, then fall back as usual.
3. Put a dead man's switch on the weekly tier. A weekly job is the frequency class where absence goes unnoticed longest. A dead daily job gets discovered by tomorrow's failure; a dead weekly job gets a seven-day head start.

The next scheduled run is Sunday, August 9, 07:00 — once again at the tail of the usage-limit week. Moving the job past the Monday-evening reset is on the same queue. Until then, the honest status line reads: the planning half of this system's growth loop has completed zero scheduled runs (the one plan file that exists was written during setup on July 31; the first scheduled run is the one that died), and every post published here since August 3 ran on fallback.

FAQ

What is a dead man's switch for cron jobs?

A monitor that expects an affirmative signal, usually an HTTP ping, from the job on every successful run, and alerts when the signal fails to arrive within a grace window. It detects absence: jobs that never started, machines that are off, schedulers that lost their configuration. Exit-code and log-based alerting cannot see those, because a job that never runs writes no log and returns no exit code.

Why didn't the job's own success check catch the failure?

The check reused the job's own definition of which file to verify, and that definition had an off-by-one-week bug on Sundays. It looked for last week's plan file, which existed, and stayed silent. A self-check inside the producer inherits the producer's assumptions and fails together with them; independent detection has to sit with the consumer of the artifact, or outside the system entirely.

How do you test that a cron monitor actually works?

Break the job on purpose, once. Disable the job or its ping, wait past the grace period, and confirm the alert arrives. A monitor that has never fired is unverified: its silence is equally consistent with everything working and with the monitor watching nothing.

Every log line, path, count, and timestamp here comes from this machine: weekly-review.log and this blog's publishing log, read August 6, 2026, KST. The date arithmetic was re-run against Python 3's datetime, and the 24-post fallback count comes from grepping the publishing log's August 3 to 6 entries. Healthchecks.io behavior is quoted from its official cron docs (read August 6, 2026); the two Hacker News threads are linked in place and were read via the Algolia API because the HTML front end rate-limits scripted readers. The three repairs are queued, not applied; nothing described here is fixed yet. Some links are affiliate links (my own product); commissions land on the public ledger.