llmfit vs whichllm: My Mac Mini Beat Both on 5 of 5 Models

I already knew how fast five Llama builds run on my Mac mini, because I measured them in September. So on 2026-10-18 I installed the two most popular "what can my hardware run" tools, llmfit 1.1.17 and whichllm 0.5.20, and asked them to predict those same five numbers. Both guessed low on all five. llmfit came in 12% to 37% under. whichllm came in 43% to 60% under, and every real number landed above the top of whichllm's own error range. For the llmfit vs whichllm question, the speed column is the part of each tool I'd trust least. The memory check is where they differ most, and whichllm got that wrong in a way that matters on a 16 GB Mac.

The machine is a headless Mac mini M4 (Mac16,10, 10 cores, 16 GB, macOS 26.4.1). The measured speeds are medians from my Ollama model ladder on this Mac mini M4, all at 100% GPU. I didn't re-run them. The only new measurements in this post are the estimators' outputs. I also ran a third tool, llm-checker 3.8.2, for contrast.

What each tool is for

llmfit (Rust, MIT, 38,031 GitHub stars on October 18) detects your RAM, GPU and backend, then scores every model in its database for fit, speed, quality and context. whichllm (Python, MIT, 6,763 stars) checks the same kind of fit, then ranks the models that fit by merged public benchmarks. Its author explained the split in the whichllm Show HN thread: llmfit answers "will this model fit in my memory?" and whichllm answers "of the models that fit, which is actually best?" Neither one runs a model to work out its speed. That was the trade-off the author accepted in the same thread: "it estimates rather than measures by design".

Installing them took two tries. whichllm requires Python 3.11 or newer, and the macOS system python3 is 3.9, so pip reported "No matching distribution found". A Homebrew Python 3.13 venv worked for both:

$ /opt/homebrew/bin/python3.13 -m venv venv
$ ./venv/bin/pip install llmfit==1.1.17 whichllm==0.5.20
$ ./venv/bin/llmfit plan meta-llama/Llama-3.1-8B-Instruct --quant Q4_K_M --context 4096 --json
$ ./venv/bin/whichllm --json -n 200 -q Q4_K_M

Predicted vs measured tok/s

I took llmfit's numbers from llmfit plan --quant so the quantization matched what I measured. Left alone, llmfit info picks "mlx-8bit" for Llama 3.1 8B and predicts 8.2 tok/s. For the three 8B rows, whichllm's numbers come from its own ranking output. Its default ranking leaves out the 1B and 3B models, so for those two rows I imported its estimate_tok_per_sec() function and ran it on my exact GGUF file sizes.

Model (Ollama, 100% GPU)	Measured	llmfit	whichllm	whichllm range	llm-checker
Llama 3.2 1B Q8_0	73.19	53.41	33.52*	20.1 to 53.6	not listed
Llama 3.2 3B Q4_K_M	46.71	41.09	26.80*	16.1 to 42.9	58
Llama 3.1 8B Q4_K_M	21.34	16.44	11.98	7.2 to 19.2	29
Llama 3.1 8B Q6_K	16.33	10.96	7.54	4.5 to 12.1	not listed
Llama 3.1 8B Q8_0	12.99	8.22	5.19	3.1 to 8.3	not listed

* whichllm's estimator run directly on my file sizes; its ranking doesn't list these models.

llmfit prints its own error margin: "Baseline error band is roughly +/-30%." The measured speed fell inside 30% above the estimate for only two of the five models, the 3B and the 8B Q4_K_M, and the 8B Q4_K_M only just made it (21.37 against 21.34). whichllm labels these estimates "medium" confidence, which in its source means 0.6x to 1.6x of the estimate. All five measured speeds were above that range.

Why they guess low: one efficiency constant

Both tools use the same physics, and the physics is right. To generate one token, the GPU has to read all of the model's weights once, so the upper limit on tok/s is memory bandwidth divided by file size. Both tools also use the right bandwidth for a base M4, 120 GB/s. The difference is the discount each one applies after that.

llmfit's estimate notes say it plainly: "GPU bandwidth roofline — 120 GB/s x 0.55 efficiency." whichllm multiplies a per-quant factor (0.55 for Q4_K_M, 0.50 for Q6_K, 0.45 for Q8_0) by a backend factor, which is 0.82 for Apple. That gives 45%, 41% and 37% of the roofline. The code comment explains the choice: "Apple's Metal kernel is behind on dequantization."

The measurements don't support that. Bytes multiplied by tok/s gives the bandwidth each model actually used on this Mac: 96.7 GB/s for the 1B, 94.3 for the 3B, then 105.0, 107.7 and 110.9 GB/s for the three 8B builds. That is 79% to 92% of 120 GB/s, and the largest file comes closest to the limit. whichllm's discount goes in the opposite direction: it gets heavier as the quant gets bigger, so its worst miss is the 8B Q8_0 at 40% of the real speed. I covered the same Q8_0 speed in Q4_K_M vs Q8_0 on this Mac. It's 12.99 tok/s, slower than Q4_K_M but faster than whichllm's 8.3 tok/s best case.

One odd detail: llmfit system printed "RAM Bandwidth: ~217 GB/s (measured)", but the estimate itself used 120. I didn't find where that 217 came from.

llm-checker misses the other way. Its speed table in apple-silicon.js is labeled "tokens/sec per B params", with 240 for a base M4. So the speed depends on parameter count, not file size. It predicted 29 tok/s for the 8B Q4_K_M (36% too fast). It also listed gemma3 at about 5.3 GB and 45 tok/s. Reading 5.3 GB per token 45 times a second needs 238 GB/s, which is twice the 120 GB/s listed in that same file.

The memory check matters more

On a 16 GB Mac, the GPU can't use all 16 GB. Metal reports a recommendedMaxWorkingSetSize, which on this machine is 12,713,115,648 bytes (11.84 GiB). Ollama treats that as a hard limit. When I raised the 8B Q8_0's context to 32,768 tokens, its footprint grew to 13 GB, only 30 of 33 layers stayed on the GPU, and speed dropped to 11.58 tok/s.

llmfit reads that limit correctly. llmfit system reports "11.84 GB GPU-available of 16.00 GB shared". The unit is wrong (the value is GiB), but the number is right. Four of its top 30 recommendations still need more than that (two at 13.18 GB and two at 12.78 GB). whichllm instead treats 16 GiB minus an 819 MB headroom as usable, which is 16.32 GB. 14 of its top 30 picks need more than 12.71 GB, including all of the top five, and every one of them is labeled full_gpu. Its number-one pick for my Mac was Qwen3-30B-A3B at Q3_K_M: a 13.36 GB file that needs 14.82 GB in total, estimated at 37.5 tok/s. Fourth was gpt-oss-20b at Q5_K_M, needing 15.83 GB, estimated at 20.7 tok/s. I haven't run either model here (gpt-oss-20b would take two hours to download on this connection). One 16 GB Mac mini owner on HN said gpt-oss-20b "eats too much ram to use for anything other than an overnight task. maybe 3tok/s."

On a Mac the fit question is the one you can't easily undo, because you can't add RAM later. That's why the Mac mini 16GB vs 24GB choice matters more than any of these tools' rankings.

Which one to use

For "will it fit on my Mac", use llmfit. It reads the Metal limit, and its errors on this machine were smaller. For speed, multiply either tool's estimate by about 1.3 for llmfit or about 2 for whichllm on a base M4. Better still, download the smallest version and time it yourself. On my machine, a 1.3 GB model took 377 seconds to download. If you're using whichllm's rankings, check every pick's memory against your Mac's real GPU limit. The tool won't do that for you.

People in the HN threads report the same thing on other hardware. In the Llmspeed Show HN (230 points on October 17), one commenter said their M2 Ultra "runs almost twice as fast" as the site's figure for an M3 Ultra. Others found a 5090 estimate too slow and a 5060 Ti range "implausible". For more detail on the 8B model itself, see Llama 3.1 8B on a 16GB Mac mini M4. For the MLX runtime llmfit defaults to, see MLX vs Ollama.

FAQ

What is the difference between llmfit and whichllm?

Both detect your hardware and list models that fit. llmfit is a Rust binary that scores models on fit, speed, quality and context, and reads the Metal GPU memory limit on a Mac. whichllm is a Python tool (Python 3.11 or newer) that ranks the models that fit by public benchmark scores. Neither runs a model to measure speed. On a 16 GB Mac mini M4 both predicted lower speeds than measured for all five Llama builds tested.

Is llmfit accurate on Apple Silicon?

On a base M4 Mac mini, llmfit's speed estimates were 12% to 37% below the measured speeds of five Llama 3.x builds, because it assumes 55% of the 120 GB/s memory bandwidth and real use was 79% to 92%. Its memory check was close, because it reads Metal's 11.84 GiB GPU working-set limit, though 4 of its top 30 picks were still slightly over it.

Why does whichllm recommend models that are too big for my Mac?

whichllm treats almost all of unified memory as usable GPU memory (16.32 GB on a 16 GB Mac), but macOS limits the GPU to its recommended working set, 11.84 GiB (12.71 GB) on a 16 GB M4. On that machine 14 of its top 30 picks, including the top five, needed more than 12.71 GB. Models over the limit spill layers to the CPU and run slower than estimated.

Estimator outputs were captured on 2026-10-18 on one headless Mac mini M4 (Mac16,10, 16 GB, macOS 26.4.1) from llmfit 1.1.17 and whichllm 0.5.20 in a Python 3.13 venv and llm-checker 3.8.2 from npm, all installed in an isolated folder and removed afterward. The measured tok/s are medians from my 2026-09-23 Ollama 0.34.2 runs at 100% GPU and were not re-measured. whichllm's 1B and 3B figures come from calling its estimate_tok_per_sec() on the exact file sizes, because its ranking doesn't list those models. Ranges, efficiency constants and the memory headroom are quoted from each tool's JSON output and source code. HN quotes are from the Algolia items API, and star counts are from the GitHub API on the same day.