Ollama vs llama.cpp Server: 4 Parallel Slots Lost to 2

This afternoon I compared Ollama and llama.cpp one request at a time and wrote "concurrent requests" under Not tested. That gap is the question people actually mean by "Ollama vs llama.cpp server": what happens when three people or three agents hit one model at once. So I ran it tonight on the same base M4 Mac mini with 16 GB, the same Llama 3.1 8B Q4_K_M blob, and Ollama 0.34.2.

My first guess was wrong. I expected OLLAMA_NUM_PARALLEL=4 to beat 2. It lost: eight simultaneous requests finished in 40.0 s with 4 slots and in 32.9 s with 2. Going to 8 slots changed nothing, and going to 16 at the default context quietly moved six layers of the model off the GPU. Everything below comes from 58 timed runs and 357 requests, each run firing N requests at once and asking for 128 tokens apiece.

What Ollama runs when you set OLLAMA_NUM_PARALLEL

Since 0.30, Ollama serves GGUF models by starting llama.cpp's own llama-server as a child process. For concurrency, the whole question comes down to which flags it passes. From ps while the model was loaded:

$ OLLAMA_NUM_PARALLEL=4 ollama serve
$ ps -axo rss,command | grep lib/ollama/llama-server
6925056 .../libexec/lib/ollama/llama-server --model .../sha256-7b064f58... \
  -c 16384 -np 4 --flash-attn auto -b 512 -ub 512 --context-shift --keep 4

The source matches. llm/llama_server.go at v0.34.2 builds -c NumCtx*numParallel and -np numParallel, and envconfig/config.go sets the default to 1, with a queue of 512 requests behind it. The runner then logs n_slots = 4, n_ctx_slot = 4096, kv_unified = 'false', so every slot keeps the full 4,096-token window.

That last detail is the one real difference from running llama-server yourself. When I passed -np 12 -c 12288 to plain llama-server for the llama.cpp parallel requests test, each slot got 1,024 tokens and a 1,502-token prompt came back as HTTP 400. Ollama multiplies the context for you, which is safer and also costs memory, as the 16-slot case shows.

You want	Ollama 0.34.2	llama-server
Parallel slots	OLLAMA_NUM_PARALLEL, default 1	-np, default auto (4 on this Mac)
Context per slot	OLLAMA_CONTEXT_LENGTH (4,096 under 24 GiB), multiplied by slots	-c divided by slots unless KV is unified
Queue when full	OLLAMA_MAX_QUEUE 512, then 503	Waits for a free slot
Per-model setting	No, one value for every model (#4894, open since 2024-06)	One model per process anyway

The sweep: slots versus requests in flight

The client starts N streaming /api/generate calls at the same instant, each with a unique first line so nothing hits the prompt cache, and divides the total generated tokens by the time until the last one finishes. Each cell ran twice, the two runs never differed by more than 3%, and the tables show their mean.

OLLAMA_NUM_PARALLEL	8 requests: total time	Aggregate tok/s	Last request's first token
1 (default)	49.9 s	20.5	43.9 s
2	32.9 s	31.2	25.0 s
4	40.0 s	25.6	20.6 s
8	40.3 s	25.5	1.3 s
16, context 2,048	40.6 s	25.2	1.7 s

The default is a queue. With one slot, every request runs at the full 21 tok/s, but the eighth person waits 43.9 s to see a single word. Eight slots fix the wait and do nothing for throughput: each stream crawls at 3.29 tok/s. What predicts the aggregate number is how many requests the runner decodes together, and the slot setting only caps that. Plotting throughput against requests in flight makes the shape obvious:

Anywhere from 4 to 8 requests in flight, the Mac does less total work than with 2. That is the same dip I traced on September 18 to one Metal kernel choice: for K-quant weights such as Q4_K, llama.cpp routes batches of 4 to 8 through mul_mv_ext, which is slower on the M4 than the path it uses at 2 or at 9 and above. Ollama bundles llama.cpp build b10969, newer than the Homebrew build I tested then, and the condition is still there in ggml-metal-ops.cpp at b10969.

This is also where the server comparison ends up. Four requests on 2 slots took 16.4 s under Ollama tonight and 16.3 s under plain llama-server on September 18. On 4 slots it was 20.1 s against 20.0 s, and nine requests on wide slots took 22.6 s against 22.4 s. Switching servers changed nothing I could measure. Changing the slot count moved the same eight-request job between 32.9 s and 49.9 s.

The 16-slot trap: six layers leave the GPU

The obvious move after that chart is 16 slots, so a burst of 9 or more jumps past the slow band. At the default context that means -c 65536 -np 16, and the runner's memory fitter printed this:

common_params_fit_impl: projected to use 13171 MiB of device memory vs. 12123 MiB of free device memory
common_params_fit_impl: cannot meet free memory target of 1024 MiB, need to reduce device memory by 2072 MiB
common_params_fit_impl: context size set by user to 65536 -> no change
load_tensors: offloaded 27/33 layers to GPU

The context was fixed because Ollama passed it explicitly, so the fitter took the only other lever and moved six layers to the CPU. Ollama logged no warning of its own; the offload shows up only in the runner output it passes through to the serve log. A single request dropped from 21.1 to 18.7 tok/s, and 16 simultaneous requests reached 56.0 tok/s instead of the 81.9 they got with OLLAMA_CONTEXT_LENGTH=2048, where the whole model stayed on the GPU. The Ollama FAQ does warn that memory scales with parallel times context length. What it can't tell you is that on a 16 GB Mac the failure mode is a slower model rather than an error.

What I would set on a 16 GB Mac

For an 8B K-quant model on a base M4, the measurements point to three settings, depending on the traffic:

• One user, or agents that run one call at a time: leave the default of 1. Extra slots only add KV memory; the runner grew from 5.5 GB at 1 slot to 6.9 GB at 4.
• A few people or a small agent fan-out (2 to 8 at once): OLLAMA_NUM_PARALLEL=2. It had the best throughput in that range and no memory pressure.
• Batch jobs that keep 9 or more requests queued: 16 slots with OLLAMA_CONTEXT_LENGTH=2048, if 2,048 tokens per request is enough. Check the log for offloaded 33/33.

Whatever you pick, verify it on the runner, not the environment you think you set. The Homebrew service sets only OLLAMA_FLASH_ATTENTION and OLLAMA_KV_CACHE_TYPE in its plist, and launchctl setenv is not a reliable way to add more:

ps -axo command | grep '[l]ib/ollama/llama-server' | grep -o -- '-c [0-9]* -np [0-9]*'
grep 'offloaded' "$(brew --prefix)/var/log/ollama.log" | tail -1   # Homebrew service; the Mac app logs to ~/.ollama/logs/server.log

What other people ran into

The same flat throughput shows up on NVIDIA. In ollama#12107, three parallel gpt-oss:20b requests took about 43 s against 30 s run one after another. A maintainer replied that concurrency "depends on hardware" and posted charts where one GPU gained aggregate speed and another didn't. On Hacker News, a comment under the 648-point thread "The local LLM ecosystem doesn't need Ollama" gives concurrency as "the quieter production reason to move" because "Ollama defaults its parallel slots low." The default is 1, so the comment is right about that. My numbers say the move itself buys nothing on this Mac: the same slot count gives the same speed in both servers, and the useful fix is one environment variable. If you're sizing the whole machine, the Llama 3.1 8B on Mac mini M4 numbers and the local LLM vs cloud cost math both assume these same streams.

FAQ

Is llama.cpp server faster than Ollama for concurrent requests?

Not on my base M4 Mac mini with Ollama 0.34.2. Ollama runs llama.cpp's llama-server underneath, and with the same slot count the two finished the same concurrent workloads within 1%. Ollama is slower out of the box only because OLLAMA_NUM_PARALLEL defaults to 1, which queues requests.

What should OLLAMA_NUM_PARALLEL be set to?

For an 8B Q4_K_M model on a 16 GB M4, 2 gave the best throughput with 2 to 8 simultaneous requests, about 31 tok/s. Values from 4 to 8 were slower, at about 25.5 tok/s. Sixteen slots reached 82 tok/s with 16 requests, but only after lowering the context to 2,048.

Does OLLAMA_NUM_PARALLEL use more memory?

Yes. Ollama passes context length times slot count to the runner, so 16 slots at 4,096 tokens reserve a 65,536-token KV cache. On a 16 GB Mac that no longer fit, and the runner moved 6 of 33 layers to the CPU without an error.

Sources and verification: all timings are my own, measured on this Mac16,10 (base M4, 16 GB, macOS 26.4.1) on 2026-09-22 between 19:35 and 20:05 KST, with other work resident and about 3.2 GB in swap. Ollama 0.34.2 from Homebrew ran as a plain ollama serve with bartowski's Llama 3.1 8B Instruct Q4_K_M; a Python client fired N streaming requests at once for 128 tokens each at temperature 0, two runs per setting (one run for 16 slots at 4,096). Launch flags come from ps and the runner's log, defaults from Ollama's source at tag v0.34.2, and the kernel condition from llama.cpp at tag b10969. The llama-server comparison figures are from my 2026-09-18 run on Homebrew build 10809. Not tested: Ollama's MLX engine, other models and quant types, long prompts, and Macs with more memory.