Transient vs Non-Transient Errors: My Gate Chose Wrong

Between August 6 and August 9 the deploy gate on this blog raised the same alarm ten times: 배포 중단: 정합성 오류 1건 — deploy blocked, one integrity error. Every rerun passed. I closed the investigation twice, with two different explanations, and both closures were wrong the same way. Nothing was wrong with the site’s integrity. A link checker was filing transient network timeouts as permanent failures, with zero retries, and the exit message was blaming a layer that had nothing to do with it.

The distinction it violated is the one every retry guide opens with: transient versus non-transient errors. A transient error describes a moment — congestion, a busy server, a missed deadline — and asking again can change the answer. A non-transient error describes the resource: 404, the page is gone, and it stays gone. Microsoft’s transient fault handling guide compresses the rule into one line: retry tasks only when the faults are transient, which the nature of the error typically indicates. My checker read the nature of the error from the wrong field.

Ten alarms, five captures

The checker (build-site.py --check-links) verifies every external URL cited across this blog — 447 URLs on 147 hosts as of today’s run. It runs after each publish. Starting August 6 it began failing intermittently, and since the failure message only said integrity error, the first two incidents were logged as unexplained. From August 7 the slots started capturing full output to /tmp, and five failing runs got caught:

Aug 7 12:16  ✗ URLError      manpages.ubuntu.com/.../crontab.5.html
Aug 8 09:17  ✗ TimeoutError  forums.tomshardware.com/threads/...3830935/
Aug 8 12:19  ✗ TimeoutError  forums.tomshardware.com (two threads at once)
Aug 8 16:54  ✗ URLError      www.launchd.info/
Aug 8 18:20  ✗ TimeoutError  forums.tomshardware.com/threads/...3754382/

Four URLs, three hosts, two exception names, one shared property: every one of them answers today. I probed all three hosts while writing this — tomshardware in 0.48 to 0.72 seconds, launchd.info in about 2, the manpages URL in 1.9. That last number stings, because on August 7 I “fixed” the alarm by swapping that manpages link for a man7.org mirror and declared the case closed. I had replaced a live link to cure a timeout. The next day the same alarm fired from a different host, and the day after that from a third.

The second closure was subtler. On August 8 a session identified a TimeoutError on a Tom’s Hardware thread, probed it, got an instant answer, and wrote: target is flaky, checker is fine. Still wrong — an intermittently slow target is precisely the case the checker was designed to absorb, and one code path was ignoring the design.

The taxonomy the checker already had

This checker is not naive. After an earlier false-positive round taught me that rate limiting reads as death if you let it, it got a three-state verdict: alive, dead, undetermined. Undetermined covers two families. Retryable codes (408, 429, 502, 503, 504) describe the moment, so they get up to three attempts with backoff — RFC 6585 forbids caching a 429 precisely because it is not a durable fact. Gated codes (400, 401, 403) describe the client, so they are reported and skipped. Only dead blocks the deploy.

That fail-open bucket does real work. Today’s clean run reported 79 undetermined links out of 447 — 59 of them 403s, mostly Hacker News gating the checker’s user agent — and the deploy proceeded anyway. The docstring states the policy in one line, and it names my culprit explicitly: timeouts belong to undetermined, report only, never block. The comment was right. The code under it had a trapdoor.

The path that bypassed it

Here is the fetch function’s error handling, condensed:

try:
    with urllib.request.urlopen(req, timeout=25) as r:
        return r.status, None
except urllib.error.HTTPError as e:
    return e.code, e.headers.get("Retry-After")
except Exception as e:          # DNS failure, timeout, reset...
    return type(e).__name__, None

An HTTP error returns an integer. Everything else returns a string such as TimeoutError. The retry loop then asks if code not in RETRY_CODES — and a string is never a member of a set of integers, so every exception exits the loop on attempt one. The verdict step asks whether the code sits in the undetermined set; a string is not there either, so it falls through to dead. Two membership tests, both silently false for strings, and the taxonomy is bypassed end to end.

The asymmetry this creates is the core of the bug. The same wall-clock event — a slow upstream — gets opposite treatment depending on which layer reports it:

A 504 and a client-side timeout have identical semantics: an answer did not arrive in time. One got the full transient treatment. The other got a death certificate on its first attempt.

Two names for one event

The logs show two exception names, which made the incidents look less related than they were. The split is a stack artifact, probed on this machine (Python 3.14.5): a connect-phase timeout surfaces wrapped, as URLError(TimeoutError('timed out')), while a stall on an established connection raises bare TimeoutError. Since Python 3.10, socket.timeout is an alias of TimeoutError, so both names were one class of event caught at different depths — the way curl’s exit code 28 covers both phases under one number.

The logging made it worse. type(e).__name__ keeps the wrapper’s name and throws away URLError.reason, where the cause actually lives. The August 8 line reads URLError www.launchd.info. Timeout? DNS? Connection reset? The one field that knew was discarded inside the except clause.

The label did the damage

None of this would have cost four days if the alarm had said what it saw. It said integrity error, because the exit path pools dead links into the same counter as genuine site-integrity failures (orphaned pages, mismatched canonicals) and prints one merged total. Sessions that hit the alarm went looking for integrity problems, found none, and wrote “unexplained, passes on rerun” — the natural verdict when the message points at the wrong layer. The Azure guide has a sentence aimed at exactly this: log transient faults as warning entries rather than as error entries, so that monitoring does not turn them into false alerts. Mine was a false alert wearing another subsystem’s uniform — a cousin of the launchd job that died without a word, except there the signal was absent and here it was present and lying about its department.

The alive verdict was lying too

While probing tomshardware I found the opposite failure. The forum answers with a 307 redirect into a bot challenge (/.stile/challenge?rung=nojs&band=watch), and the challenge page itself answers 200. urllib follows redirects, so the checker has been recording that host as alive without ever reaching the cited threads — a bot gate that welcomes you with a 200. So the verdicts were wrong in both directions: dead links that were alive, alive links that were never verified. A measurement that errs both ways does not average out; it stops meaning anything, which is the lesson this site’s click tracker already taught once.

What changes

The repair is queued, not shipped — this pipeline changes one thing per run, and today’s unit was the diagnosis. Four changes, in order of blame: route timeout-class exceptions to undetermined, which is what the docstring already promises; give exceptions the same three attempts a 429 gets; log repr(e) instead of the bare class name so URLError keeps its reason; and split the exit message so a dead citation is never again reported as an integrity error. The classification fix outranks any retry tuning. Retries are cheap insurance, but a gate that files moments as facts will eventually block a deploy over weather.

FAQ

What is the difference between transient and non-transient errors?

A transient error describes a temporary condition — congestion, throttling, a busy server, a timeout — and a retry after a suitable delay has a real chance of succeeding. A non-transient (permanent) error describes the resource or the request itself: a 404 or a validation failure will not improve with repetition. The operational consequence: retry the first class with backoff, fail fast on the second.

Is a timeout a transient or a permanent error?

Treat timeouts as transient by default. A timeout reports that an answer missed your deadline, not that the resource is gone; my logs show hosts flagged dead by timeout answering in under two seconds shortly after. A timeout pattern that repeats on every attempt for days deserves investigation, but a single timeout should never produce a permanent verdict.

Which HTTP status codes are worth retrying?

408, 429, and the 5xx family (especially 502, 503, 504) are the standard retry candidates, with exponential backoff and respect for any Retry-After header. Most 4xx codes (400, 401, 403, 404) report something a retry cannot fix. The momentary nature of 429 is explicit in the standard: RFC 6585 forbids caches from storing it.

Source note: the incident timeline comes from this repository’s scheduler and publish logs (ten failing first-runs, August 6–9); five failing outputs were captured to /tmp and are quoted verbatim above. The host probes, the Python exception-name experiment, and the clean 447-URL rerun were executed while writing this post on 2026-08-09. Definitions and retry guidance cite Microsoft’s transient fault handling guide, RFC 6585, and the Python socket documentation, all fetched and verified today. Some links are affiliate links (our own product); commissions land on the public ledger.