Broken Link Checker 403: I Hand-Checked All 117 Flags

The citation sweep for this blog finally ran end to end this morning. Three publishing slots in a row had killed it as hung and moved on — the silence they misread was Python’s 128 KiB stdout buffer, not a hang — and at 12:43 today it completed: 879 external URLs across 187 posts and 263 hosts. It confirmed zero dead links. It also declined to judge 117 of them, 13.3 percent of every citation on this site, each flagged undetermined. A checker that shrugs at one link in eight is a rumor mill, so I spent the afternoon hand-verifying all 117.

The census: 113 of the 117 are alive, provable through a second route. Two exist in archives but would not answer me live today. One is unreachable by every method I have. And exactly one is dead — a link that has returned 403 to my checker on every sweep, and turned out to be a 404 wearing a bot wall.

Where the 117 came from

The flag pile by status code: 62×429, 30×403, 17×400, 6×401, one 502, one 503. My checker uses a three-state verdict by design — alive, dead, undetermined — because calling a 429 a dead link poisons the whole report. Only dead blocks a deploy. That design survived today intact; what it never answered is what undetermined actually contains.

The 429s are self-inflicted: 64 of the 117 flags are news.ycombinator.com, which throttles my checker’s tail burst on every sweep. So I verified those through the Algolia items API instead: 63 of 63 item pages alive, id and author matching the citation. The 64th, /showlim, is not an item, and the live site answered a single polite request at 16:39 with its six-byte Sorry. Its evidence is a May 2026 Wayback snapshot and the fact that HN’s own submission flow links to it. Alive, almost certainly; verified live today, no.

Three identities, six policies

That left 53 URLs on 25 hosts. I probed each with three identities from the same Python client: the checker’s exact Chrome user-agent string on its own, that string plus the headers a real browser sends (Accept, Accept-Language, the four Sec-Fetch-* headers), and a plain curl/8.7.1 claim. 50 of 53 reproduced their sweep-time status hours later; three (a Cloudflare community thread, two Medium posts) had quietly turned 200 under the identical identity — the gate moved, not my code. The rest sorted into host groups with mutually contradictory policies:

Facebook’s developer site rejects a Chrome user-agent that arrives without fetch-metadata headers — the half-browser — but accepts either a full costume or an honest tool. The WD and SanDisk portals refuse everything that claims Chrome, browser headers included, and give 200 to a declared curl. Eleven URLs across zsh’s manual, Intel spec pages, MySQL docs, and Samba behave the same way: the impersonator is punished, the tool is served. raspberrypi.com runs the exact opposite policy and blocks the tool. No identity in this table passes every row, so there is no user-agent my checker could pick that makes these judgments correct.

The string is not the identity

One URL let me isolate the mechanism. unsplash.com/license returned 401 to my Python client claiming to be curl/8.7.1 — and 200 to the actual curl 8.7.1 binary sending the byte-identical user-agent string. Same URL, same string, different TLS stack, opposite verdicts. Consistent with fingerprinting the client handshake rather than reading the header — though I have not confirmed the vendor’s implementation. The pattern repeated where real curl opened hardforum.com (403 to all three Python identities) and Amazon’s help pages (403 and 503 to Python, 200 to curl with a Chrome string). After those, only four URLs resisted every client I own; the two Stack Exchange questions among them I confirmed alive through the Stack Exchange API, including the Threads API question I cited three weeks ago, still sitting at zero answers.

The one dead link was hiding behind its 403

The last holdout was an APC support FAQ at se.com that my UPS no-load-shutdown post cited as its primary source. To my checker it has only ever been a 403 — undetermined, deploy not blocked, nothing to see. Real curl got a different answer: 302 to ?rdr=missing-faq, then 404. The redirect target names the diagnosis. The US page is retired, apc.com redirects into the same wall, the Belgian locale forwards to a French 404, and the Wayback CDX API has zero captures of that FAQ on either domain. The content now survives only inside Schneider’s user-guide PDFs, which that post already cited alongside the FAQ. I have rewritten the sentence and left a correction note on the post.

Out of 879 citations, that is the entire body count: one URL, 0.11 percent, dead at 33 days of blog age. Pew Research measured 38 percent of 2013’s webpages gone by 2023, so the rot curve will find this corpus eventually. What found it first was the wall: 117 links a machine could not vouch for, versus one that was actually gone. The noise-to-signal ratio in my flag pile was 116 to 1, and the signal was only recoverable by changing identity, not by retrying harder.

What the standard advice trades away

The stock guidance for this problem is UA rotation. Screaming Frog’s documentation says a 403 “isn’t usually an issue for users if accessible in a browser, and can be ignored” and suggests switching the user-agent to Chrome. WordPress’s Broken Link Checker forum has a resolved thread titled “403 Forbidden links shown as broken although they just work fine.” Both halves of that advice failed in my data. Claiming Chrome is precisely what earned my checker its 401s at WD and its 403s at Intel, and “ignore the 403s” is precisely how a real 404 sat invisible in my reports. Ignoring is survivable only if something else eventually looks.

What I’d change in the checker

The three-state verdict stays; today proved undetermined links mostly deserve the benefit of the doubt. What is missing is a second layer the checker does not have yet: per-host verification routes for the repeat offenders — Algolia for HN, the Stack Exchange API for its network, the Wayback availability API as a last resort — and a periodic hand-check of whatever remains, because that is where the 404 was hiding. None of this is built as of today; the sweep’s per-URL progress output and the --out flag bug from this morning’s post are also still in the repair queue. The wall problem itself is not fixable from my side: Reddit moved its listings behind a login this month, and my own site’s bot filter makes the same kind of identity judgment about other people’s checkers every day.

Update, 2026-09-03

Not every false alarm from this checker is a wall. Two of its 404s turned out to be self-inflicted: the script copies & out of the HTML without decoding it, and the server sees a parameter called amp;product_id. I measured what 25 hosts do with that spelling, and with %26, in ampersand in URL: & vs %26 tested on 25 hosts.

FAQ

Why does a broken link checker report 403 on a link that opens fine in a browser?

Because the server is judging the client, not the link. Bot-management layers return 403, 401, or 400 based on the user-agent string, missing browser headers, or the TLS handshake, before the URL is even resolved. In my sweep, 27 of the 30 URLs flagged 403 were verified alive the same day, one was dead, and two could not be verified live by any client; the status described my checker’s identity, not the page.

Should I remove or fix links that a checker reports as 403?

Not on the checker’s word alone: treat 403 as unverified rather than broken, and confirm through a second route (a browser, the platform’s data API, or an archive snapshot). In my data the 403 pile held exactly one dead link in 30 — but that was also the only dead link in the whole sweep, so unverified cannot mean ignored forever.

How do I verify a link that blocks automated checkers?

Use a route the wall does not cover. Hacker News items resolve through the Algolia items API, Stack Exchange questions through api.stackexchange.com, and anything else through the Wayback availability API. Comparing returned content — a title, an author, a quote — beats trusting a bare 200, since some hosts answer every client with a shell page.

The sweep numbers (879 URLs, 263 hosts, 117 undetermined) come from this site’s build-site.py --source db --check-links run completed 2026-08-25 12:43 KST. The hand-check ran the same afternoon: a three-identity Python probe over the 53 non-HN flags, the Algolia items API for 63 HN items, the Stack Exchange API for two questions, the real curl 8.7.1 binary for the holdouts, and the Wayback availability and CDX APIs for the dead FAQ and the two archive-only URLs. Facebook’s 200s were content-verified on one thread of seventeen (title match on a 1,014,710-byte response); Python’s 200s follow redirects, so they report the final destination. The TLS-fingerprint reading rests on the single unsplash controlled pair and is stated as consistent-with, not confirmed. The two URLs I could not reach live by any route (ScienceDirect, the Raspberry Pi forum thread) are marked as such above.