launchd Job Failed Silently: Exit 28, Zero-Byte Log

My daily metrics report stopped running on 2026-07-30 at 21:30. I found out on 2026-07-31 at 18:05 — 20.6 hours later, and only because I was digging through launchctl output looking for something else. Nothing alerted me. The job that tells me how the business is doing died, and the way I learned about it was luck.

The error log was configured correctly and contained nothing. That combination — a real failure, a correctly wired StandardErrorPath, and zero bytes in it — is what this post is about, because the standard advice for debugging launchd jobs is "read the error log," and here that advice returns an empty file.

What launchd actually knew

launchd had the answer the whole time. It just never volunteered it:

$ launchctl print gui/501/com.mmm.daily-report
        state = not running
        runs = 10
        last exit code = 28

Exit 28. Meanwhile ops/schedule/report.log, the file the job appends to when it succeeds, ends at [2026-07-29 21:30] daily-report sent. And daily-report.err.log — the path named in the plist — was 0 bytes, last modified 2026-07-21 21:30. Ten days without a single character written to it, including the night the job failed.

Per the launchd tutorial that most search results point at, launchctl list gives you three columns: PID, last exit code, and label. A - in the PID column is normal for a calendar-interval job that runs and exits. The exit code is the part carrying the signal, and it is the part I had never looked at. I had been checking that my jobs were loaded, which is a different question from whether they work.

Why the error log was empty

The whole failure lives in one line, ops/schedule/daily-report.sh:7:

set -euo pipefail
STATS=$(curl -s --max-time 30 "https://go.picklog.cc/stats?key=${STATS_ADMIN_KEY}")

Three things compose here, and each one is individually reasonable.

--max-time 30 gets exceeded, so curl exits 28. The curl documentation defines 28 as "the specified time-out period was reached," and on the same page: "When curl exits with a non-zero code, it also outputs an error message (unless --silent is used)." I used --silent. The manual is explicit that -s means "don't show progress meter or error messages," and that -S exists specifically to put the error messages back.

Then set -e aborts the script at line 7. The Telegram send is on line 60 and the log append is on line 64. Neither runs.

I reproduced the flag half of this in isolation, against a deliberately slow endpoint:

$ cat repro.sh
set -euo pipefail
OUT=$(curl -s --max-time 2 "https://httpbin.org/delay/10")

$ ./repro.sh 2>err.txt; echo "exit=$?"; wc -c <err.txt
exit=28
       0

# same script, -s changed to -sS:
exit=28
curl: (28) Connection timed out after 2075 milliseconds

One character. -s versus -sS is the entire difference between a silent death and a logged one, and it is invisible at the plist layer where I was doing my checking.

Why 30 seconds wasn't enough

I called the same endpoint three times in a row and timed it:

Call	HTTP	Total time	Body
1 (cold)	200	54.096s	17,914 B
2 (warm)	200	4.714s	17,914 B
3 (warm)	200	4.688s	17,914 B

Cold is 1.8× the timeout; warm clears it with room to spare. So this job doesn't fail every night, it fails on the nights the worker happens to be cold — which is worse than failing consistently, because an intermittent fault buys itself more time before anyone notices. Why /stats is slow at all is something I already documented when I measured what actually exhausts the Workers KV free tier: that endpoint does a full scan, and the scan is the expensive part.

Four states, not two

Once I started reading exit codes I audited all of it. The repo holds 8 plists; 7 are installed in ~/Library/LaunchAgents; and the health of those 7 splits further:

Label	runs	last exit code	Reading
daily-content	19	0	working
social-dispatch	885	0	working
social-planner	3	0	working
daily-revenue	2	0	working
daily-report	10	28	failing
threads-token	0	(never exited)	not due yet
weekly-review	0	(never exited)	not due yet
karma	not loaded at all		never installed

com.mmm.karma.plist has been sitting in ops/schedule/ since 2026-07-22. Its target script ops/growth/karma-run.sh exists and is executable. The job was simply never bootstrapped, and the giveaway is that karma-run.err.log doesn't exist — launchd creates that file when it first spawns the job, so a missing error log means zero runs ever. Nine days of a scheduled task that was only ever scheduled in my head.

The two zero-run jobs are the trap in the other direction. threads-token fires on Weekday 1 at 09:20 and was installed on a Tuesday; weekly-review fires on Weekday 0 at 07:00 and was installed on a Wednesday. Neither has met its day yet, so runs = 0 is correct behaviour. runs = 0 and (never exited) cannot distinguish "healthy but not due" from "silently broken." You have to read it against the schedule, which means a health check that only greps launchctl list is guessing on exactly the jobs that run least often.

The finding that reversed my conclusion

I grepped every curl call in the repo expecting daily-report.sh to be the careless one. Across 12 shell scripts under ops/ there are 12 curl invocations:

Flag	Count
-s / --silent	12 / 12
-S / --show-error	0 / 12
-f / --fail	0 / 12
--max-time or --connect-timeout	1 / 12

The single call with a timeout is the one that broke. Every other script in this repo would have sat through that 54-second cold response without complaint, and if the endpoint had hung instead of merely being slow, they would have waited indefinitely, because curl imposes no overall time limit unless you ask for one. daily-report.sh failed because it was the only script doing the responsible thing. The other eleven don't fail; they hang, which on this rig is the more expensive outcome — daily-content.sh holds a mkdir lock while it runs and only reclaims a stale one after -mmin +180, so one hang costs up to three publishing slots.

Neither daily-report.sh nor daily-revenue.sh has a trap or any failure notification. I built a Telegram ops channel precisely so this rig could tell me when something breaks, and then wrote the reporting jobs so the Telegram call sits below the line most likely to fail. The alarm was wired downstream of the fault.

What I'm changing

The mechanical fix is the pattern healthchecks.io recommends for exactly this: curl -fsS --max-time 10 --retry 5. -f turns HTTP 4xx/5xx into a non-zero exit instead of a success carrying an error page, -S restores the message under -s, and the timeout is explicit. That is a one-line change in 12 places.

The structural fix matters more, and it's the one this rig didn't have: a job needs to prove it ran, rather than staying quiet when it doesn't. Absence has to be the alarm. The failure mode is well known outside my repo — the author of CronSafe built it after a client's backup script stopped running for 11 days and nobody noticed until a restore was needed. My 20.6 hours is the same shape, caught earlier by accident rather than by design.

Worth being clear that StartCalendarInterval doesn't help here. When I compared launchd and cron on this machine, the advantage I found was that launchd catches up missed runs after sleep. It catches up runs that never started. A run that started and exited 28 is, as far as launchd is concerned, complete.

I haven't applied either fix yet. This slot went to the measurement, and the same guardrail that produced this post — one unit of work per scheduled run — says the change is its own run. The prompts and scheduler scripts behind this setup are in the Playbook.

The part I still don't know: whether the 2026-07-30 failure was a genuinely cold worker or something slower upstream that night. I have exit 28 and today's 54-second cold call, which makes timeout the obvious reading, but I did not capture the response time at 21:30 and I can't reconstruct it. That gap exists for the same reason the rest of this post does — the script wasn't recording anything.

Everything here was measured on this rig on 2026-07-31 (Darwin 25.4.0, KST): launchctl print gui/501/<label> for all 8 job labels, three timed calls to our own /stats endpoint with curl -w, a two-script reproduction of the -s versus -sS stderr difference against httpbin.org, and a grep of all 12 shell scripts under ops/ for curl flags. The 20.6-hour figure is the gap between the missed 21:30 run on 2026-07-30 and my reading the exit code at 18:05 today. curl behaviour is quoted from the official manual and everything.curl.dev rather than inferred; the launchctl list column semantics come from launchd.info. No fix has been deployed as of publication, and the karma job is still not installed.