llama.cpp KV Cache Quantization: q4_0 Fits 128K on 16GB

Three days ago I ran Llama 3.1 8B on this 16 GB Mac mini and stopped at a number I did not like: at a 64K context the default f16 KV cache pushed 3.5 GiB of other processes into swap and decode fell from 21 to 14.6 tokens per second. The model's full 128K window needs 16 GiB for the cache alone, which is the whole machine. llama.cpp KV cache quantization is the usual answer: two flags, -ctk and -ctv, that store the cache in 8 or 4 bits instead of 16. A week ago a Hacker News commenter who runs them wrote that "I settled on q8_0 because it is the only way to fit the model + 100k tokens into 24GB VRAM" and asked for a benchmark of what that costs.

So I ran the same model with f16, q8_0, q4_0 and a mixed cache, and measured memory, speed and how far each one's predictions drift from f16. q8_0 barely moved the predictions and added no swap at 64K. q4_0 cost 1.26% in perplexity but was the only type to run the full 128K window here, at 20.8 tokens per second, and the mixed pairing gets flash attention refused by the Metal backend.

What the two flags change, in bytes

For every token in the context, Llama 3.1 8B keeps a key and a value vector in each of its 32 layers, 8 key-value heads of 128 dimensions each. That is 65,536 numbers per token, and -ctk and -ctv decide what each number costs. The llama.cpp server documentation lists nine allowed types and a default of f16, two bytes per number. q8_0 packs 32 numbers into 32 one-byte integers plus a two-byte scale, 34 bytes per block or 8.5 bits each. q4_0 packs the same 32 into 16 bytes plus the scale, 18 bytes or 4.5 bits.

The weights are another 4.58 GiB, and Metal on this machine grants one process recommendedMaxWorkingSetSize = 12713.12 MB, about 11.8 GiB, the same ceiling I found when running a local LLM on the 16 GB Mac mini the first time. On paper that puts f16 out of room between 32K and 64K, q8_0 just over the line at 128K, and q4_0 comfortably under it. The flags go on the command line like this; I used -fa on everywhere, because Ollama's FAQ says the cache is only quantized with flash attention enabled, and keeping it on for f16 too means the cache type is the only thing that changes.

llama-completion -m Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  -c 65536 -fa on -ctk q8_0 -ctv q8_0 \
  -p "List three facts about launchd." -n 48 --temp 0

Memory: what each cache type buys on a 16 GB machine

I re-ran the context sweep from three days ago with each cache type: the same 8-token prompt, 48 tokens out, -c set to 32K, 64K and 128K, reading the process's peak resident memory and the system's swap before and after.

At 32K all three fit with no new swap, and the gaps between them match the byte table to within 0.1 GiB. At 64K the f16 process reached 11.99 GiB, the edge of what Metal grants, and macOS moved 2.22 GiB of other work into swap while decode fell to 17.6 tokens per second. q8_0 at 64K used 8.90 GiB, added 0.01 GiB of swap and decoded at 20.5.

At 128K the two quantized types split. q4_0 ran the model’s full window in 9.22 GiB, added no swap and decoded at 20.8 tokens per second, the same speed as at 32K. q8_0 did not run. The process reached 11.88 GiB and Metal refused the first batch:

error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
llama_decode: failed to decode, ret = -3

That is the byte table arriving as an error message: 4.58 GiB of weights plus 8.5 GiB of cache is 13.1 GiB against an 11.8 GiB grant. It failed in four seconds instead of swapping, which I prefer, but it means q8_0’s ceiling on this machine sits somewhere between 64K and 128K, and I did not bisect it.

Speed: a little slower to read a prompt, a little faster once the cache is full

With an empty cache, quantizing it costs 2% of prompt speed at q8_0 and 7% at q4_0, and about 2% of decode. With 16,384 tokens already in the cache the order flips: both quantized caches decode 8% faster than f16, because every generated token reads the entire cache and theirs is half or a quarter of the bytes. Filling the 16K took 107 to 110 seconds whichever type I used.

Quality: q8_0 moves almost nothing, q4_0 moves as much as a weight quant

To measure what the commenter was losing I used the method in llama.cpp's perplexity documentation. Run WikiText-2 through the model once with an f16 cache and save the probability it gives every possible next token. Then run the same text with a quantized cache and measure how far those distributions moved (KL divergence), how much perplexity changed, and how often the most likely token stayed the same. I used eight chunks of 2,048 tokens; llama.cpp scores the second half of each, so 8,192 tokens were judged. The weights are Q4_K_M in every run, so the cache is the only variable.

q8_0 is about as close to free as this measurement can show: a mean divergence of 0.00005, and perplexity up 0.0002, far inside its own ±0.151 error bar. For scale, the same documentation's scoreboard for Llama 3 8B puts the divergence from quantizing the weights to Q8_0 at 0.001355, 27 times what the q8_0 cache cost here, and the Q4_K_M weights I run at 0.031. That table uses the previous model generation, a CUDA backend and the whole test set, so I compare orders of magnitude, not decimals.

q4_0 is a real cost. Its mean divergence is 0.0105, 211 times q8_0's, and the most likely token changed about once every 23 tokens. That is roughly what the scoreboard charges for Q5_K_M weights (0.0108). Because the reference was this same Q4_K_M model with an f16 cache, it is drift added on top of whatever the weight quantization already cost.

Update, September 15: I have since measured the weight side on this machine, with Q8_0 as the reference because F16 does not fit. Q6_K drifted 0.0028 and Q4_K_M 0.0187 over the same 8,192 tokens, so the Q4_K_M weights move the predictions more than a q4_0 cache does. The numbers are in Q4_K_M vs Q8_0 on a 16 GB Mac mini.

Mixing 8-bit keys with 4-bit values is the usual compromise, and on this Mac it has a cost the other two do not: its quality run took 36.55 seconds per pass where matched types took 10.36, three and a half times slower. The reason is in llama.cpp's Metal backend. At the commit my build came from, ggml-metal-device.m answers "can you run flash attention?" with if (op->src[1]->type != op->src[2]->type) return false;, so any K and V pair that differ is handed to another backend. The line is unchanged on today's master. One llama-bench run shows the bill: prompt processing at 143 tokens per second instead of 229, and decode at 17.0 instead of 21.1. The same check lists the types Metal does accept, and iq4_nl, which the command line allows, is not one of them.

Its quality lands between the other two: a mean divergence of 0.0016, the top token unchanged 98.4% of the time. Keeping the keys at 8 bits removed about 85% of the drift that all-4-bit caused. I did not run the reverse pairing, so that is not proof that keys are the sensitive half, only that this pairing works, and on a Mac it costs the speed.

What other people report, and what my test cannot answer

The strongest warning I found comes from one Hacker News commenter, twice. In August they wrote that "even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks" and is "basically unnoticeable for simple and short tasks" (comment); in June, the same account described "a substantial degradation in long context performance with any of these quants" (comment). That is one person's experience, not two reports, and neither comment names a model or a measurement. On the other side, an owner of a single RTX 3090 runs OLLAMA_KV_CACHE_TYPE=q8_0 with a 180,000-token context and says it "fits in 23GB" (comment).

Ollama's FAQ adds a caveat that matters for other models: those with a high GQA count "may see a larger impact". Llama 3.1 8B shares each key-value head across 4 query heads. Qwen2-7B, the model the FAQ names, shares each across 7, and in llama.cpp's 4-bit KV cache discussion a user reported that "On Qwen2-7B, q4_0 produces weird results. q8_0 is ok".

What I would set on a 16 GB Mac

q8_0 for both keys and values, with -fa on, whenever the context goes past 16K. It halves the memory, gives 8% of decode back once the cache fills, and barely moved the predictions. Below 16K there is nothing to win on this machine: f16 fit with zero swap three days ago, and today even 32K added none. q4_0 is for the case where nothing else fits, which on this machine means the full 128K window: q4_0 ran it at 20.8 tokens per second where q8_0 ran out of memory, and I would check its output on my own task first, since its drift is about what the scoreboard charges for Q5_K_M weights. I would not run mixed types on a Mac while that Metal check stands. And if what you want is f16 at 64K, that is not a flag, it is the 16 vs 24 GB decision.

What this test cannot settle is whether q8_0 hurts long agentic sessions the way the commenter above describes. My longest scored window was 16K tokens of Wikipedia text, positions 8,192 to 16,383, and there q8_0 did not drift further (a mean divergence of 0.000053 against 0.000050 at 2K) while q4_0 did: its mean rose 22% to 0.0129, and its worst 0.1% of tokens moved 55% more. A 100K-token coding session is a different workload, and I have not run one.

FAQ

How do I enable KV cache quantization in llama.cpp?

Pass -ctk and -ctv with a cache type, for example -ctk q8_0 -ctv q8_0, together with -fa on for flash attention. The same flags work in llama-server, llama-cli, llama-completion, llama-bench and llama-perplexity. Allowed types are f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0 and q5_1, and the default is f16. In Ollama the equivalent is the OLLAMA_KV_CACHE_TYPE environment variable with flash attention enabled.

How much memory does a q8_0 or q4_0 KV cache save?

q8_0 stores each cached value in 8.5 bits and q4_0 in 4.5 bits, against 16 for the default f16, so q8_0 uses 53% of the f16 cache and q4_0 uses 28%. For Llama 3.1 8B that is 131,072 bytes per token of context in f16, 69,632 in q8_0 and 36,864 in q4_0: a 64K context costs 8 GiB, 4.25 GiB or 2.25 GiB, and the full 128K window costs 16, 8.5 or 4.5 GiB.

Does KV cache quantization hurt output quality?

At q8_0 the measured change is close to nothing: on Llama 3.1 8B Q4_K_M over 8,192 WikiText-2 tokens, a q8_0 cache gave a mean KL divergence of 0.00005 from the f16 cache, perplexity 0.003% higher, and the same top token 99.66% of the time. q4_0 is a real cost: mean KL divergence 0.0105, perplexity 1.26% higher, and the same top token 95.6% of the time. Scored deeper into a 16K context, q8_0 stayed at 0.00005 while q4_0 rose to 0.0129, and models with more query heads per key-value head can be affected more.

Update 2026-09-18: Serving several requests multiplies the cache, and setting -np yourself also splits -c between slots unless you add -kvu: a 12-slot server with -c 12288 rejected a 1,502-token prompt. Details in llama.cpp parallel requests.

Sources and verification: every speed, memory, swap and divergence figure is my own reading on this Mac16,10 (base M4, 10-core GPU, 16 GB, macOS 26.4.1) on 2026-09-15 between 18:17 and 18:43 KST, using Homebrew llama.cpp 0.4.0 build 10809 (commit 5266f24da), bartowski's Meta-Llama-3.1-8B-Instruct Q4_K_M GGUF (4,920,739,232 bytes, the same file as my 12 September post) and -fa on for every cache type. Quality runs used llama-perplexity with --kl-divergence-base on the WikiText-2 test file from llama.cpp's own download script. Memory is /usr/bin/time -l maximum resident set size plus sysctl vm.swapusage read before and after each run, with 11.3 GiB of other work resident and 1.34 GiB already in swap when I started; a model download was also writing to disk at about 3.5 MB/s throughout. The byte counts per token come from the model's config.json and the ggml block sizes. The Metal check is quoted from the source at my build's commit and confirmed unchanged on master the same day. Hacker News comments were read through the Algolia API; I could not access Reddit. Not tested: tasks longer than 16,384 tokens, agentic coding sessions, and any model other than Llama 3.1 8B.