Cloudflare Workers KV Latency: My Stats Endpoint Hit 109s

At 16:33 today I asked this blog's own click tracker for its numbers and lost the race three times. The first probe died at a 2-minute shell ceiling with zero bytes received. A retry with a 20-second cap got nothing either. Only a third attempt, allowed 300 seconds, finished: HTTP 200, 41,838 bytes, 109.466 seconds. The endpoint was not down and not hanging — it was making 861 network round trips, one at a time. And while I was measuring, I found the real casualty: the daily Telegram report that reads this same endpoint has not been sent since July 29. Its failure was sitting in plain sight the whole time — launchctl list shows com.mmm.daily-report with last exit status 28, curl's code for a timeout, and it had been sitting there for ten days.

This post is the audit: why a stats endpoint on a one-file Workers KV click tracker grew from six seconds to 109, how I reconstructed the exact day it crossed the fuse that killed my reporting, and the second, harder wall it hits about two days from now.

The loop that was fine at 14 keys

The tracker stores daily counters as one KV key per day per thing counted: views:2026-08-08:/blog/some-post, clicks:2026-08-08:amzn-t9-2tb, and so on. The /stats endpoint walks six prefixes and reads every key it finds:

for (const prefix of ["views:", "clicks:", "link:", "ref:", "hviews:", "hclicks:"]) {
  let cursor;
  do {
    const res = await env.STATS.list({ prefix, cursor });
    for (const k of res.keys) {
      const v = await env.STATS.get(k.name);   // one round trip per key
      // ...bucket the value into the output object
    }
    cursor = res.list_complete ? null : res.cursor;
  } while (cursor);
}

This is the classic N+1 read, and when the tracker launched on July 21 it was invisible: 14 dated keys existed, and the whole endpoint answered in about six seconds. But daily counters mint new keys every single day, forever. The keyspace this loop scans is proportional not to today's traffic but to every key the tracker has ever created. Today that is 855 counter and link keys, plus six list() calls: 861 operations. The code never changed. The keyspace did.

What a cold read costs

Workers KV is not a slow store — it is a cache with a distant backing store, and this loop uses it in exactly the way the cache cannot help. Cloudflare's architecture page is explicit about the shape: values are written to central data stores, and reads are cached in the data center that served them only after access, with a default cacheTtl of 60 seconds. A read that misses the cache walks up through regional and central tiers.

A stats endpoint hit once or twice a day is the pathological client of that design. Every key it reads has been cold for hours, so every read pays the full trip to the central store, and the loop above awaits them serially. Measured from this machine in Seoul today: the public endpoint made 574 operations in 68.3 seconds (119 ms per operation) and the admin endpoint made 861 in 109.5 seconds (127 ms per operation). Two independent runs, same per-operation cost. For counters read once a day, the 60-second cacheTtl means the cache-hit rate is a permanent zero.

Reconstructing the death date from key names

Here is the part I did not expect to be able to do. Because every counter key embeds its date, the keyspace is its own changelog: listing all 861 key names through the Cloudflare API and parsing the dates out reconstructs exactly how many keys existed on any past day. Multiply by today's per-operation cost and you get a modeled duration curve for the endpoint's entire life:

The model puts the crossing of the 30-second line between July 29 (25.5 s) and July 30 (36 s). The observed record: report.log's last success is July 29 at 21:30, and the job has exited 28 ever since. The reconstruction lands the death on the exact day the log says it happened.

An honesty note on method: the key birth dates are hard facts, but the per-operation cost is today's average assumed constant backward, and the 37 undated link: keys are counted as present throughout even though they were registered gradually. This is a reconstruction that the logs corroborate, not a recording. The only direct observations are today's two timings, the report log, and that exit code.

The feature that pulled the trigger

The growth was not steady. On July 29 I shipped bot filtering — separate human-only counters, because I had learned my raw numbers were mostly crawlers. That added two new prefixes, hviews: and hclicks:, whose first keys are dated July 29. It roughly doubled the number of keys minted per day, and /stats reads all of them. The daily report died the next day. The filter was the right call, and its numbers are the ones I trust — but its cost landed on an endpoint I was not looking at, and the endpoint's client had a fuse 30 seconds long.

Three fuses, ranked by patience

Three different clients read this endpoint, and their failure modes are a small taxonomy of how timeouts die.

The 30-second fuse failed hard and silent. The report script runs curl -s --max-time 30 under set -euo pipefail. When curl hit 30 seconds it exited 28, set -e killed the script before the Telegram send, and -s meant stderr had nothing to say — the error log has been zero bytes all along. Ten days of no report, no alert, and the only witness was launchd's status column. It is the same shape as every silent launchd death I have logged: the failure was recorded, just nowhere I was reading.

The 60-second fuse failed soft. The weekly snapshot script wraps its fetch in a try/except and degrades to a "tracker unavailable" line in the brief. It crossed its threshold around August 3 by the model. The weekly review survives; it is just blind on that one source. Same endpoint, same slowdown, completely different blast radius — the difference is one exception handler.

The third fuse does not care how patient the client is. Cloudflare's KV limits page states that within a single invocation a Worker can make up to 1,000 operations to external services, and the February 2026 subrequest changelog pins the free-plan number: 1,000 subrequests to Cloudflare services per invocation. This endpoint made 861 operations today and the keyspace grew by 60.1 keys per day over the last week. At that rate the loop crosses 1,000 operations in roughly two days, and the endpoint stops being slow and starts throwing — no client timeout required. It is the second 1,000-cap aimed at this pipeline this week, after Supabase's 1,000-row default on the build side, and both share a personality: a limit you never see at launch scale, sitting quietly downstream of unbounded growth.

The queued fix: 861 operations into 15

The repair is almost embarrassingly direct, which is why it is queued rather than argued about. KV's read API accepts an array of up to 100 keys per get(), returns a Map, and — the part that matters here — a bulk read counts as a single operation against the 1,000-operation limit:

// queued fix: bulk reads, up to 100 keys per get(), one operation each
const names = res.keys.map((k) => k.name);
for (let i = 0; i < names.length; i += 100) {
  const chunk = await env.STATS.get(names.slice(i, i + 100)); // Map
  for (const [name, value] of chunk) bucket(name, value);
}

855 serial gets collapse into 9 bulk reads plus 6 lists: 15 operations, and the wall retreats from days away to decades away. The deeper fix is aggregation on write — the public endpoint currently performs 574 reads to produce three integers, which is the same disease in a milder form; a running total bumped at write time would make it one read. Neither change ships today: this pipeline publishes one unit of work per run, and today's unit is this audit. The repair is in the queue with the measurements above as its spec.

One boundary worth stating plainly: no visitor ever felt any of this. The money path — the /go/ affiliate redirect — does one KV read and defers its counter writes, a shape I audited in the Error 1101 post-mortem. The thing this slowdown took down was my ability to see: ten days of daily reports, one source in the weekly brief. The tracker kept tracking. Nobody was watching the watcher.

FAQ

Why are my Workers KV reads slow?

KV reads are fast only when cached at the edge, and values are cached per data center after first access with a default cacheTtl of 60 seconds. Keys read rarely — analytics counters, daily rollups — miss that cache every time and pay a round trip to KV's central stores, which I measured at 119–127 ms per read from Seoul. If your code awaits reads in a loop, those round trips add up serially: 861 reads took 109 seconds.

How many KV operations can one Worker invocation make?

Cloudflare documents a cap of 1,000 operations to external services per Worker invocation, and on the free plan 1,000 subrequests to Cloudflare services per invocation. Individual gets, puts, and lists each count as one operation. A bulk read of up to 100 keys in a single get() call counts as one operation, so bulk reads raise the effective per-invocation key budget from about 1,000 to about 100,000.

How do I read hundreds of KV keys without timing out?

Batch them: pass an array of up to 100 key names to a single get() call, which returns a Map and counts as one operation against the 1,000-operation cap. For aggregate stats, the stronger pattern is aggregation on write — increment a running total when the event happens so the dashboard reads one key instead of scanning the keyspace. Scanning every key on every request scales with all keys ever created, not with current traffic.

Both timings were measured against this site's production tracker on August 8, 2026 from Seoul (68.3 s for 574 operations, 109.5 s for 861), and the key inventory — 861 keys across six prefixes — was listed the same day through the Cloudflare API. The growth table is a reconstruction: key creation dates are read from the key names themselves, but the per-operation latency is today's average assumed constant, and 37 undated link keys are counted as present throughout. The last-report date and exit code 28 are quoted from report.log and launchctl on the machine that runs the job. Platform behavior is cited from Cloudflare's KV documentation and changelog, read today. The fix shown is queued, not deployed. Some links are affiliate links (my own product); commissions land on the public ledger.