Curl Exit Code 28: The Timeout That Stayed Silent 10 Days

My nightly metrics report last succeeded on July 29 at 21:30 KST. It died the next night, and I diagnosed that death quickly: on July 31 I published a post-mortem of the launchd side — the zero-byte error log, the -s flag that ate curl's error message, the set -e that killed the script before its own alerting could run. That post even named the one-character fix. Then the fix went into a repair queue, the publishing loop moved on, and the job died again every single night: ten misses as of last night. Earlier today a latency audit of the endpoint it calls dated the exact cause. This post is the half I never wrote down: what curl exit code 28 actually is, what --retry really does with a timeout (measured, with one genuinely sneaky result), where launchd hides the number, and what a diagnosed-but-unfixed failure costs per night.

What exit code 28 actually is

Curl's manual defines exit 28 in one line: Operation timeout. The specified time-out period was reached according to the conditions. The libcurl error list names it CURLE_OPERATION_TIMEDOUT, and its neighbor is the useful contrast: error 7 is "Failed to connect() to host or proxy" — a refusal. 28 is not a refusal. The connection worked, bytes may already have been flowing, and then a clock you set ran out. When you see 28, the first question is not "is the server down" but "which of my own deadlines fired."

Two flags arm that clock. --connect-timeout covers only the connection phase. --max-time covers the whole transfer — the manual says it is "useful for preventing your batch jobs from hanging for hours on a single file transfer." My script had the second one:

set -euo pipefail
# ...
STATS=$(curl -s --max-time 30 "https://go.picklog.cc/stats?key=${STATS_ADMIN_KEY}")

Thirty seconds was generous when that stats endpoint answered in six. By July 30 the endpoint's own keyspace had grown until a cold response took longer than the fuse — the KV latency post-mortem reconstructs the growth day by day, puts today's cold read at 109 seconds, and dates the fuse crossing to the night of July 29 or 30. The last successful report in the log is July 29, 21:30. The arithmetic and the log agree to the day.

Reproduced tonight, eight days after diagnosis

Because the fix never shipped, the failure is still available on demand. Tonight, by hand:

$ time bash ops/schedule/daily-report.sh
bash ops/schedule/daily-report.sh  0.02s user 0.01s system 0% cpu 30.172 total
$ echo $?
28

30.172 seconds of wall clock, exit 28, no stderr, no Telegram message, no log line. The mechanics of that silence are the July 31 post's territory, so one paragraph here: -s without -S suppresses curl's own curl: (28) Operation timed out line; set -e treats the failed command substitution as fatal — bash's manual is explicit that for an assignment-only statement "the exit status of the command is the exit status of the last command substitution performed" — so the script dies at line 7, and the Telegram send at line 60 plus the log append at line 64 are never reached. Every alarm sat downstream of the one line most likely to fail.

Where the evidence lived: 28, or 7168

One detail the first post-mortem missed, and it will save you a confused half hour. The same launchctl asked two ways gives two different numbers for the same death:

$ launchctl list | grep daily-report
-	28	com.mmm.daily-report

$ launchctl list com.mmm.daily-report
{
	"LastExitStatus" = 7168;
	...
};

7168 is 28 × 256. The table view decodes the exit code for you; the detail view hands you the raw wait status, where the exit code lives in the high byte, wait(2)-style. If you script a health check against the detailed output — which is the view that offers per-job key-value pairs, so it is the natural target — grepping for = 28 finds nothing while the job is failing with exit 28 nightly. Divide by 256, or read the table.

What --retry does with a timeout

Curl's manual defines exactly which failures --retry acts on: "Transient error means either: a timeout, an FTP 4xx response code or an HTTP 408, 429, 500, 502, 503 or 504 response code." A timeout qualifies — exit 28 is retriable out of the box, with backoff starting at one second and doubling. My script requested zero retries, which is also the default.

But the manual's wording for --max-time ("the entire operation") left me unsure how the two flags compose: with retries enabled, is the time budget shared across attempts or granted per attempt? Measured against my own slow endpoint:

$ time curl -s --max-time 5 --retry 1 -o /dev/null "https://go.picklog.cc/stats?key=..."
curl ... 0% cpu 11.467 total
$ echo $?
28

11.467 seconds = a 5-second attempt, a 1-second backoff, and a second 5-second attempt. Each retry gets its own --max-time budget; use --retry-max-time to cap the whole envelope. And the honest caveat: retry only rescues flaky. This endpoint is not flaky — it needs 109 cold seconds every time, so three retries at a 30-second fuse would have produced four timeouts and the same exit 28. Retry papers over variance, not arithmetic.

A second experiment produced the sneakiest result of the day. Against a public test endpoint that was itself misbehaving, --max-time 2 --retry 2 returned exit 0 in 4.5 seconds: the first attempt timed out, and the retry received a fast error page — an HTTP failure, which bare curl counts as a completed transfer. Without --fail, a retry can convert an outage into a green exit code with garbage in your variable. If the next line of your script is json.loads, that trade is strictly worse than the timeout: exit 28 at least told the truth.

The fix that stayed in a queue

Here is the uncomfortable part. The July 31 post identified that changing -s to -sS — one character — would have logged every death. I did not deploy it. This site runs on a one-unit-of-work-per-run publishing loop, repairs go into a queue, and the queue kept losing to the next post for eight straight days while the thing the queue was about kept dying nightly. A diagnosed failure with an unshipped fix behaves identically to an undiagnosed one; the diagnosis only changes how embarrassing the eventual write-up is. The queued repair, still queued as I publish:

STATS=$(curl -sS --fail --max-time 120 --retry 2 "$URL") || {
  rc=$?
  "${MMM_ROOT}/ops/telegram/notify.sh" "daily-report FAILED: stats fetch, curl exit ${rc}"
  exit "$rc"
}

-sS restores the error line, --fail refuses to call an HTTP error page a success, and the || block does two jobs at once: it disarms set -e for this one command, and it gives failure an exit path that speaks. That last property is the general rule this incident keeps teaching: under set -e, any notification placed below a fallible line fires only on success. Alerting has to be structurally attached to the failure — or inverted entirely, like a dead man's switch that alarms on absence. The Telegram ops channel this report feeds worked the whole ten nights; nothing upstream was shaped to reach it.

And the timeout itself? Raising --max-time to 120 buys headroom, but the endpoint that needs 109 of those seconds is the actual defect — its repair (bulk KV reads, 861 round trips down to about 15) is queued in the KV latency post, and yes, I notice which queue that one is in. Exit 28 was never the disease. It was the only symptom honest enough to write itself down every night.

FAQ

What does curl exit code 28 mean?

Exit code 28 is CURLE_OPERATION_TIMEDOUT: a time limit you set — usually --max-time or --connect-timeout — expired before the transfer finished. It is not a connection refusal (that is exit 7); the server may be up and responding, just slower than your deadline. Check which timeout flag your command sets and how long the request actually needs before blaming the network.

Does curl --retry retry after a timeout?

Yes. Curl's manual classifies a timeout as a transient error, alongside HTTP 408, 429, 500, 502, 503 and 504, so --retry N retries exit-28 failures with exponential backoff. Each attempt gets its own --max-time budget; cap the total with --retry-max-time. The default retry count is zero, so a bare curl call never retries.

Why does launchctl show exit status 7168 instead of 28?

The detailed view, launchctl list <label>, reports LastExitStatus as a raw wait status, where the exit code is stored in the high byte: 7168 = 28 × 256. The plain launchctl list table shows the decoded code. Divide the detailed value by 256 to recover the exit code a shell would report.

Every number here was gathered on August 8, 2026 on the machine that runs the job: the reproduction (30.172 s, exit 28), both launchctl views (28 and 7168), the retry timing experiment (11.467 s), and report.log's last success line of July 29. The launchd-side mechanics were first published in the July 31 post-mortem linked above; this post extends it rather than re-deriving it, and admits the eight days in between. Curl semantics are quoted from the curl manual and libcurl error list, read today; bash semantics from the bash manual on this machine. The exit-0-after-retry result came from a flaky public test endpoint and is described, not reproducible on demand. Both fixes shown remain queued, not deployed. Some links are affiliate links (my own product); commissions land on the public ledger.