Hacker News Search API: 39% of My Top Hits Were Noise

The tool that picks what this blog writes about reads Hacker News through the Algolia search API. This morning I audited what it had been feeding the planner: 17 of the 43 top stories it reported had no match for the query in their title at all, and the worst query returned a solar eclipse map as its second-best result for mac mini.

Nothing was broken. Every request returned HTTP 200. The API did its job correctly and I was the one holding it wrong, in two separate ways.

What the harvester actually does

The relevant code is about twenty lines — build a window, ask for 30 stories, re-sort them:

# ops/analytics/demand.py - cmd_hn
since = int((datetime.now(timezone.utc) - timedelta(days=args.days)).timestamp())
url = ("https://hn.algolia.com/api/v1/search?tags=story&hitsPerPage=30"
       f"&query={urllib.parse.quote(q)}&numericFilters=created_at_i>{since}")
...
hits.sort(key=lambda h: -(h["points"] + h["comments"]))
out[q] = {"window_days": args.days, "n": len(hits), "top": hits[:12]}

Four queries run every week: claude code, ai agent, cloudflare workers, mac mini. That top[:12] is what the weekly review reads when it decides which subjects are worth a post.

The first problem is arithmetic

The response carries an nbHits field. I had never looked at it. Running the URL above for the seven days ending 2026-08-14:

query	nbHits in window	received	never seen
claude code	138	30	108
ai agent	238	30	208
cloudflare workers	7	7	0
mac mini	18	18	0

The harvester never paginates, so two of four queries drop 78–87% of the window on the floor. That bug is mine and it is boring. The interesting part is what the surviving 30 were.

The second problem is that the title is not the only thing being searched

Every hit carries a _highlightResult object naming the field the match landed in. I had been ignoring it. The three stories the tool reported as top results for mac mini:

points	title	matched in
527	Show HN: Needle2: 14MB agentic LLM for phones, wearables…	story_text
198	Open-source interactive map for the Aug 12 total solar eclipse	title: ['mac'], url: ['mini']
47	Launch HN: Keet (YC S24) – An app to create video courses	story_text

The eclipse map is the one that gave it away. Its title contains map, one character substitution from mac, and Algolia's typo tolerance accepted it. The word mini came from the URL. The other two are Show HN posts whose body text happened to mention a Mac mini — a legitimate match against a field I did not know was searchable.

You can map the searchable set by feeding restrictSearchableAttributes names and reading which ones bounce. Exactly five are accepted — title, url, author, story_text, comment_text — and everything else returns 400 attribute X is not in searchableAttributes setting. That matches the index configuration in the hn-search repo, the Rails app behind the search, archived and read-only since 2026-02-10. The API reference at hn.algolia.com/api is client-rendered and says none of this.

How badly it lands depends entirely on the query

Counting the tool's top[:12] for each query, and marking every story whose title does not literally contain the query words:

query	no title match
claude code	0/12
ai agent	0/12
cloudflare workers	6/7
mac mini	11/12
total	17/43 = 39.5%

This is the shape worth remembering: not a uniform error rate you can discount. A distinctive multi-word phrase like claude code is perfectly clean; two short common words are almost entirely noise. If you had only ever tested the API with a specific product name you would never see this.

The API had already ranked it correctly

Here is the part that reframed the whole thing. The archived README gives the index ranking as ["typo", "proximity", "attribute", "custom"] with a customRanking of ["desc(points)", "desc(num_comments)"]. Points are the last tie-breaker, applied inside a relevance tier, not across tiers. So the API deliberately puts a 2-point story with a clean title match above a 527-point story that only matched body text.

Then my code re-sorts the whole list by points + comments and flattens those tiers:

API rank	rank after our re-sort	points	title match level
5	1	527	none
11	2	198	partial
13	3	47	none

Every false positive I complained about was already demoted. My sort function dredged them back up. The noise did not come from Algolia; it came from me applying a second ranking on top of a ranking I had not read.

Two knobs, and the two indexes agree once you set them

/search and /search_by_date reported different totals for the same window — 18 versus 12 for mac mini. I assumed different corpora. It is typo tolerance, configured differently on the two indexes:

setting	/search	/search_by_date
default	18	12
typoTolerance=false	6	6
restrictSearchableAttributes=title	4	1
both	1	1

They converge exactly. The same holds for a single word: mac gives 1,028 versus 826 by default and 200 versus 200 with typo tolerance off — roughly 83% of hits for a three-letter query are typo matches.

The fix is not free. Restricting to title means every query word must appear in the title, which drops real results: cloudflare workers falls from 7 to 2, losing Unifying Workers AI and AI Gateway into a Single AI Control Plane and Building my own project management tool on Cloudflare. claude code falls from 137 to 85, and 7 of the dropped items have a partial title match. You are trading false positives for false negatives. The better lesson is the one from the ranking table: keep the API's ordering and filter it, rather than re-ranking it yourself.

The 1,000-hit wall arrives as an empty page

If you do paginate, there is a ceiling. It is uniform across page sizes and always returns HTTP 200:

hitsPerPage	page	offset	hits returned
1000	0	0	1000
1000	1	1000	0
500	2	1000	0
100	9	900	100
100	10	1000	0
10	100	1000	0

This is Algolia's paginationLimitedTo, defaulting to 1,000 because "a lower limit can help prevent full data scraping". You cannot raise it from the client: paginationLimitedTo=5000 returns 400 Unknown parameter, and /api/v1/browse is a 404.

The response does explain itself, but in a place that is easy to miss. Alongside nbHits: 0 and an empty hits array, inside a 200, there is a message field: "you can only fetch the 1000 hits for this query". I missed it on my own first probe because I was printing len(hits). A client that checks status codes, or loops until it gets an empty page, sees nothing wrong. That is the same shape as the silent short pages we hit listing Workers KV keys, and of a piece with a success exit code covering a run that did nothing. Issue #230 reported this in August 2022 — "this limit is not made explicit on the HN Search API reference" — and it is still open, under a repository that has since been archived.

Everything validates strictly except the one thing that matters

Bad parameters fail loudly and usefully. numericFilters=points>>5 returns 400 Invalid syntax for numeric condition. hitsPerPage=abc returns a 400 naming the expected range. An unknown searchable attribute returns a 400 telling you it is not in the settings.

But tags=stroy — the typo you would actually make — returns HTTP 200 with nbHits 0. So does tags=storys. The tag value is a filter, and a filter matching nothing is not an error. Valid values, for reference: story (1,917,740), comment (227,165), front_page (43), poll (22).

Two more things worth knowing. The params field echoes back what the server actually ran, which is how you discover it injects advancedSyntax=true on both endpoints (this is why a quoted "mac mini" phrase works) and hitsPerPage=20 on /search_by_date. And there are no rate-limit headers at all — the documented 10,000 requests per hour per IP is invisible from any response, the same blind spot as the missing Retry-After when Hacker News throttled our link checker.

What I would tell someone starting today

Use /search for relevance and /search_by_date for a window, and do not re-sort either by points. Read nbHits before you trust your page. Read _highlightResult to see which field matched — it is the only thing in the response that tells you why a hit is there. If your query is short or common, set restrictSearchableAttributes=title and accept the false negatives. And filter server-side: numericFilters=points>10 takes ai agent from 237 hits to 21 without any of this mess.

In fairness, the API is excellent at the job it is built for. While I was measuring all this, its newest indexed story was 14 items and 2.8 minutes behind the official Firebase API's maxitem. Freshness is not the problem. The problem was that I built a planning input on top of search semantics I had never read, and then sorted over them.

I have not fixed demand.py yet — this slot is one unit of work, and the change touches how the weekly review chooses subjects, which I would rather not alter in the same pass that discovered the bug. The three candidate fixes are in the notes. Until then, the tool's output is one input among several to a planner that already runs on fallbacks, which is the only reason 39.5% noise produced bad suggestions rather than bad posts.

Every number here was measured on 2026-08-14 from Seoul against the live hn.algolia.com/api/v1 endpoints, using a seven-day window (created_at_i>1786077085) and the four queries this site's planner actually runs. The truncation, top-12 and ranking tables come from the verbatim URL in ops/analytics/demand.py; the typo-tolerance and pagination tables are controlled sweeps varying one parameter at a time. The searchable-attribute list was derived by probing 13 field names and recording which returned 400, then checked against the archived repository's index configuration. The 10,000 requests per hour figure is documented, not measured — I did not test it. Counts on Hacker News change continuously, so the window totals will not reproduce exactly. The fix is not applied.