Link Checker False Positives: 429 Isn't a Dead Link

On 2026-07-28 my deploy stopped because of a rate limit on a documentation site I do not own. The build's citation checker walked all 43 external URLs quoted across this blog, got HTTP 429 from docs.postgrest.org, recorded it as a dead link, and exited non-zero. Nothing on my site had changed. Nothing on PostgREST's site had changed either.

This morning I ran the identical check and all 43 links came back alive. Then I fetched the exact URL that had "died" twelve times in a row, as fast as Python would issue the requests: twelve 200s, no Retry-After header, Server: cloudflare. The failure would not reproduce, which turned out to be the most useful fact about it.

A checker that cries wolf gets muted

The reason a link checker false positive matters more than it looks is what happens next. My checker's output feeds straight into the deploy gate in ops/build-site.py: dead links become errors, errors call sys.exit, publishing stops. So the first time it flagged a link I knew was fine, my instinct was to reach for the flag that turns the check off. That instinct is the whole problem. A checker with a known false positive rate gets ignored, and an ignored checker is worse than no checker, because no checker at least does not tell you that you are safe.

So the question is not "how do I silence 429". It is "what is my checker actually claiming, and is that claim true".

Sorting the URLs turned the crawl into a burst

The original loop was one line of ordinary tidiness:

for url, slug in sorted(urls.items()):

Sorting made the report readable. It also grouped the requests by host, because the scheme and hostname are a prefix of the URL string, so every URL on the same domain sorts into a contiguous run. Across this blog's 43 citations there are 22 distinct hosts, and they are not evenly spread: six point at code.claude.com, five at developers.cloudflare.com, four at github.com, two at docs.postgrest.org. Sorted order fires those runs back to back with no gap between them. A cosmetic choice about output ordering had quietly become a request-rate policy.

The checker already had a retry. From the day it was added it slept six seconds and tried once more on a 429, which is why I assumed the case was handled. Retrying does not help when the burst is the cause: you replay one request into a host that is still annoyed about the previous five, and six seconds is a guess with no relationship to the server's actual window.

The spec already says 429 is not a verdict

Before writing any code I went back to the definition. RFC 6585 section 4 defines 429 and includes a line that settles the design question: "Responses with the 429 status code MUST NOT be stored by a cache." A status that may not be cached is one the spec refuses to treat as a durable fact about the resource. It describes a moment, and specifically a moment about the requester's own behaviour.

The same section makes Retry-After optional, saying a response "MAY include a Retry-After header indicating how long to wait before making a new request". That matched my measurement, since the Cloudflare-fronted response carried no such header, so any backoff depending on the server to name its window needs an answer for when it stays silent.

A 404 says the resource is gone. A 429 says you asked too fast. Recording both in the same column loses the distinction that matters.

Accepting 429 as success is the wrong shape

This is a known sore spot in the established tools, so I read how they handle it. The lychee rate-limits documentation is direct about the odds: "checking many links from a single website, chances are you will get rate limited at some point." Its advice is to lower concurrency, to set --max-retries 0 because retrying makes rate limiting worse, and as a fallback to pass --accept 200,429. The matching report, lychee issue #367, was filed in October 2021 by a user seeing intermittent 429s from github.com even with a token and concurrency reduced to ten, and one remedy floated there is giving a busy host its own schedule — the same diagnosis I reached from the sorted-order bug.

The --accept 200,429 escape hatch is the part I did not copy. It promotes 429 to success, so an endpoint that returns 429 forever, which is an endpoint no reader of mine can open, sails through the check. I wanted a third answer, not a wider definition of success.

Three states instead of two

Alive and dead cannot express "I could not find out". So I added the missing state and let only one of them stop a deploy:

# 429/503 describe our request rate or the server's mood, not the resource.
# RFC 6585: a 429 response must not even be cached.
UNDETERMINED_CODES = {408, 429, 502, 503, 504}

# hosts round-robin so consecutive requests go to different domains
by_host = {}
for url in sorted(urls):
    by_host.setdefault(urllib.parse.urlsplit(url).netloc, []).append(url)
ordered = []
while by_host:
    for host in list(by_host):
        ordered.append(by_host[host].pop(0))
        if not by_host[host]:
            del by_host[host]

for url in ordered:
    for attempt in range(3):
        code, retry_after = fetch(url)
        if code not in UNDETERMINED_CODES or attempt == 2:
            break
        wait = 4 * (2 ** attempt)          # no Retry-After: back off
        if retry_after and str(retry_after).strip().isdigit():
            wait = min(int(str(retry_after).strip()), 30)
        time.sleep(wait)
    if isinstance(code, int) and 200 <= code < 400:
        continue
    (undetermined if code in UNDETERMINED_CODES else dead).append((code, url, slug))

Only dead reaches the deploy gate. Undetermined URLs print under a ? marker stating that the verdict could not be reached and the deploy is proceeding anyway. Three retries with growing gaps replaced the single six-second nap, and Retry-After is honoured when the server sends it, capped at thirty seconds so a hostile value cannot park the build.

Testing it against a server that misbehaves on purpose

The original bug will not reproduce on demand, so waiting for another live 429 was not a test plan. I ran the checker against a local HTTP server with three endpoints instead, which makes every path deterministic. This is the same reflex behind the idempotency guards for LLM cron jobs here: if the failure is timing-dependent, build the timing.

The run took 4.6 seconds, /throttle received exactly three requests, and the list handed to the deploy gate contained the 404 and nothing else. Against the real citation set the check now reports 43 links across 22 hosts, all alive, in 32.9 seconds against 45.9 before. Interleaving the hosts made it faster as well as calmer, because the run is no longer serialised behind one domain's patience.

What I am still getting wrong

Undetermined is a state I have to actually read. If docs.postgrest.org genuinely disappears next month, the checker prints a ?, the deploy goes out, and a broken citation sits in a post until I notice. The honest fix is to escalate a URL to dead once it has been undetermined for several consecutive runs, which requires remembering previous runs and the checker remembers nothing, and I am also still calling 403 dead even though some hosts serve it to any non-browser client.

The general lesson is narrower than "handle rate limits". An automated check should be able to say it does not know. Everything I run unattended, from Claude Code on a schedule to the build that turns Supabase rows into this site, produces verdicts nobody reads in real time, and a verdict I cannot trust costs more than a missing one because I act on it once and then stop acting on it forever. The 204 that looked like a successful delete was this mistake pointed the other way: a status code answering a different question than the one I was asking.

The same failure mode reached our click counter, in a form that was harder to see because nothing errored. Every hit on a /go/ redirect was recorded as a click, so crawler sweeps arrived as a confident number rather than an unknown one, and the affiliate click tracker bot traffic that produced a 35.5x gap against the merchant report looked exactly like traffic until I checked its shape.

FAQ

Does HTTP 429 mean a link is broken?

No. A 429 means the server is rejecting your request rate, not that the resource is missing. RFC 6585 states that 429 responses must not be stored by a cache, which is the specification's way of saying the status describes a passing moment rather than a durable fact about the URL. A link checker should treat 429 as an unresolved check and retry later, while 404 and 410 are genuine evidence that the target is gone.

Should a link checker fail CI on a 429?

No, and it should not treat 429 as success either. Failing the build on a rate limit produces false alarms that train you to bypass the check, while accepting 429 as a valid status lets a permanently throttled endpoint pass forever. Report it as a third state meaning the verdict could not be reached, visible in the log without blocking the deploy, and escalate to a failure only if the same URL stays unresolved across several consecutive runs.

How do I stop my link checker from getting rate limited?

Spread the requests across hosts instead of walking the URL list in sorted order. Sorting groups every URL from the same domain into a contiguous run, so one host absorbs the whole burst. Bucket the URLs by hostname and take one from each bucket in turn, which sends consecutive requests to different domains. Then honour the Retry-After header when the server sends it, with a capped exponential backoff as the fallback, since RFC 6585 makes that header optional and many servers omit it.

The 429 that started this was logged in this project's notes on 2026-07-28, and the checker it broke was added the same day with its retry already in place. The 43 URLs, the 22 hosts and the per-host counts come from querying this site's live post bodies today; the twelve consecutive 200s and the absent Retry-After header are from a probe against the URL that had been reported dead. The 45.9 and 32.9 second timings are before-and-after runs of the same command over the same citations, and the three-endpoint test is a local server written for this post reporting its own output. The 429 semantics are quoted from RFC 6585 rather than recalled, and the lychee behaviour is from its documentation and issue tracker. Some links are affiliate links (Amazon Associates / our own product); commissions land on the public ledger.