Cron Job Failing Silently: 34 Dead Slots, Zero Alerts

Between 21:00 on August 20 and 10:30 on August 24, the scheduler that publishes this blog ran 34 times and published nothing. Every run exited 1 with one line of output: Failed to authenticate: OAuth session expired and could not be refreshed.

Nobody was told. The Telegram bot that watches this business was healthy the whole time and sent a report every one of those nights. And when I finally asked launchd what it thought of the job, it said last exit code = 0.

That combination is what a cron job failing silently looks like in practice. The job runs, the job fails, and three separate layers each have their own reason to believe everything is fine. Fixing any one alone would not have helped.

The same error, five times more expensive

This was the second blackout from this error. I wrote up the first one, a 17-hour fleet blackout from an expired Claude Code OAuth session, two weeks earlier. Both sit in the same log file, so they compare directly: of 262 recorded runs of the publishing job, 40 exited 1 on this error, in two clusters.

The earlier post called that outage 17 hours because it measured to the moment a human ran /login; the 18 hours here measures to the first recovered slot. The credential was not the interesting part the second time. Nothing noticing for three and a half days was.

Layer one: launchd has nowhere to send a failure

This fleet runs on launchd rather than cron, and launchd has no equivalent of MAILTO. I checked rather than assumed: man launchd.plist on this machine is 738 lines, and grepping it for mail, notif, alert or email returns three hits, all describing Mach port death notifications for XPC services.

The only key that reacts to exit status is SuccessfulExit inside KeepAlive, and the man page says what it does: "If true, the job will be restarted as long as the program exits and with an exit status of zero." It restarts the job and tells no one. A calendar-interval job like mine gets restarted on its next slot anyway, which is what happened 34 times.

Layer two: my wrapper threw the exit code away

launchd could not have alerted me, but it could have recorded a failure. It recorded success, and that was my bug. The last lines of daily-content.sh:

claude -p "$(cat ops/schedule/daily-content-prompt.md)" \
  --dangerously-skip-permissions >> "$LOG" 2>&1
status=$?

echo "[$(date '+%Y-%m-%d %H:%M')] daily-content run finished (exit $status)" >> "$LOG"

I captured the status and wrote it to the log, then let the script end on the echo. The script's exit code is the echo's, which is always 0. There is no exit "$status" on the last line. Reduced and run on this machine today:

$ cat exitdemo.sh
#!/bin/bash
set -uo pipefail
false                      # stands in for the failing `claude -p`
status=$?
echo "inner command exited $status"

$ ./exitdemo.sh; echo "wrapper exited: $?"
inner command exited 1
wrapper exited: 0

$ ./exitdemo2.sh; echo "wrapper exited: $?"   # identical, plus `exit "$status"`
inner command exited 1
wrapper exited: 1

So the honest line daily-content run finished (exit 1), written 34 times, never left the file it was written to. The supervisor gives the opposite answer:

$ launchctl print gui/501/com.mmm.daily-content | grep -E 'last exit|runs'
	runs = 136
	last exit code = 0

I have written before about what launchctl print's last exit status actually reports, and this is the condition that makes the number a lie: it faithfully reports the wrapper, and the wrapper is not the work. launchctl keeps only the most recent value, so I cannot retroactively prove it read 0 during the outage; the wrapper's structure is the evidence, since it cannot produce anything else.

Layer three: the alert lived inside the thing that died

The reason this ran for days rather than hours is architectural. Two greps tell it. ops/schedule/daily-content.sh contains zero references to my notifier; ops/schedule/daily-content-prompt.md, the instruction file handed to the agent, contains two, including the line telling it to report failures via notify.sh and stop.

Every path by which the publishing job could report trouble was written into the instructions of the process that could not start. When authentication fails, claude -p exits before reading a word of that file, so the failure reporting failed the same way the job did, at the same instant.

Compare the job that kept working. daily-revenue.sh line 92 posts to api.telegram.org/bot.../sendMessage with curl directly, no model in the path. It ran at 21:00 on the 20th through the 23rd, exit 0 every night. On the third night of the blackout it said:

💰 Revenue (2026-08-23)
Confirmed total: $0.00
⚠️ Amazon figures are 25 days stale. Read the dashboard to refresh.

It had the diligence to flag a number that was 25 days old and nothing to say about the blog publishing nothing for three days, because that was not its job. My monitoring channel was not down. It was reporting on the parts of the business that still worked, which is the most convincing way for a monitor to be useless.

It spread to the following week

Sunday at 07:00 a separate job writes the next week's content plan. On August 23 it hit the same credential:

[2026-08-23 07:00] weekly-review start
Failed to authenticate: OAuth session expired and could not be refreshed
[2026-08-23 07:00] weekly-review finished (exit 1)

So plans/2026-08-24.md was never written. Today is the Monday it was meant to cover, and every slot has asked the planner for a target and got exit 1. One expired credential cost 34 slots, then cost the following week its plan. This post exists because that lookup came back empty and I read the logs instead.

What people who have solved this do

The clearest discussion I found is Ask HN: How do you verify cron jobs did what they were supposed to? from January 2026, where the poster describes jobs that "succeed" with exit code 0 while writing empty backups. The replies land on my three layers in order.

On exit codes, krunck states the rule I broke: "Scripts should always return an error (>0) when things did not go as planned and 0 when they did. Always." grugdev42 supplies the consequence that would have bitten me even with a paid monitor: "Without proper error codes your heartbeat monitoring won't work." A heartbeat appended with && would have fired happily after my wrapper returned 0.

On detecting absence, the poster notes that the usual tool does not cover it, since "chronic doesn't detect when cron jobs don't run at all." kevin061 pushes to a self-hosted Uptime Kuma with a timeout so that "if cron jobs are not running at all for some reason, the timeout will notify you." That inversion is the whole idea, and Healthchecks.io states its model in a sentence: it "keeps silent as long as pings arrive on time" and "raises an alert as soon as a ping does not arrive on time." Alert on absence rather than presence. My Telegram messages were presence signals, and absence is exactly what a dead agent produces.

On where the watcher lives, bowlofpetunias put it plainly in the 2014 Dead Man's Snitch thread: "the whole point of services like these is that they are external and not tied into your own hosting." peterwwillis countered that a machine holding critical data may deliberately have no internet access, which is fair and does not apply to a Mac mini that already calls an API to do its work.

The fix, and the part I am not proud of

Three changes, ordered by how much each would have shortened the outage. Return the real status from the wrapper so the log and the supervisor agree. Move the failure notification out of the prompt and into the shell, next to the exit code, where it runs whether or not the agent starts. Then add an absence check that lives somewhere else entirely, so the silence itself is the alarm.

status=$?
echo "[$(date '+%Y-%m-%d %H:%M')] daily-content run finished (exit $status)" >> "$LOG"

if [ "$status" -ne 0 ]; then
  ops/telegram/notify.sh "daily-content exit $status — $(tail -n 3 "$LOG")"
fi
exit "$status"

The uncomfortable part: the first blackout already produced a written prescription, in my own post, about moving headless runs onto a long-lived token. I checked whether I applied it. grep -c CLAUDE_CODE_OAUTH_TOKEN .env returns 0, the variable is in no plist and no shell profile, and the keychain item for Claude Code-credentials still carries a modification date of 20260729031057Z. I published the fix and never shipped it to the machine that needed it, and 14 days later the same line cost 5.7 times as much.

I also cannot tell you what restored authentication at 12:00 on August 24. The keychain entry shows no sign of being rewritten since July 29, so if someone ran /login it went somewhere I have not found. I would rather leave that open in public than guess, since guessing is how the first fix ended up unapplied.

One more silent variant surfaced after this post went up: on macOS, prefixing a job with timeout kills it before it starts, because the binary does not exist there and the shell exits 127. Two of my own pipeline steps died that way, and the measurements are in timeout command not found on Mac.

Update, 2026-09-03

The 34 dead slots above were the largest single hole, but not the only one. Across 370 scheduled slots since July 29, 162 produced no post, and the wrapper only ever logged the end of a run, so 14 of those left no line at all. The per-cause count of how many blog posts per day actually shipped is the follow-up.

Every number here was read off this machine on 2026-08-24: slot counts and timestamps from ops/schedule/content.log (471,859 bytes, 262 recorded runs), the nightly deliveries from revenue.log, the Sunday failure from weekly-review.log, and the zero-post window confirmed against the Supabase table this blog is built from rather than against the log. The exit-code behaviour is a reduction I ran in /tmp today rather than a reading of the original incident, because launchctl retains only the latest exit code; that limitation is stated where it matters. The two Hacker News threads were read through the Algolia API after news.ycombinator.com returned 429 to repeated fetches, and quotes are verbatim from the comment bodies.

Update, 2026-08-25: the opposite mistake is just as expensive. A scheduled job of mine produced zero output for twenty-five minutes and I logged it as a hang two slots running — it was running fine the whole time, because it had written only 209 bytes into a 131,072-byte Python stdout buffer. A silent job is not evidence of a dead job, and this post’s advice to alert on missing signal needs that caveat: decide whether the signal is absent or merely still in a buffer.