I run open-source models on my desktop, a machine with an RTX 5070 Ti and 16GB of VRAM. Everyone starts with Ollama because it's one command. The enthusiasts swear by llama.cpp. The serious deployment people run vLLM. I knew the engines perform differently, but I wanted real numbers: same model, same GPU, how big is the gap actually?
So I spent a few evenings benchmarking all three. Same prompts, same harness, same machine, one engine on the GPU at a time. I started with two models and realised partway through that the lineup was missing a third case, so I added it later. Glad I did, because it changed the conclusion.
And the models are the real trick. Three of them, covering the three situations you can be in with a 16GB card:
One honest detail before the numbers. My desktop session stayed running the whole time, which eats about 1GB of VRAM. That's the box as I actually use it, and that 1GB ends up playing a real part in the story.
Everything runs. Nobody crashes. And yet:
| gpt-oss-20b | TTFT | Prefill (long) | Decode | 8 clients |
|---|---|---|---|---|
| Ollama | 0.25 s | 4,160 tok/s | 166 tok/s | 152 tok/s |
| llama.cpp | 0.07 s | 7,490 tok/s | 226 tok/s | 505 tok/s |
| vLLM | 0.25 s | 2,010 tok/s | 33.5 tok/s | 129 tok/s |
Same weights, same quantization, three wildly different outcomes. And yes, the datacenter engine really did 33 tokens a second. Seven times behind llama.cpp, on identical weights.
Here's what happened. vLLM couldn't load the model at all at first. Three out-of-memory crashes, because 13.8GB of weights plus its working space didn't fit next to my desktop's 1GB. The only way to get it running was telling it to park 3GB of the weights in system RAM. And the catch is that vLLM still does all its math on the GPU, so those parked gigabytes travel across the PCIe bus for every single token. VRAM moves data at roughly 900GB a second. PCIe moves it at 64. Generating tokens is basically a memory bandwidth game, and vLLM was forced to play it across the slow bus.
Hold that thought, because round three shows what this engine does when nothing is choking it.
llama.cpp beating Ollama by 36%, despite Ollama wrapping the same engine? Two boring reasons: a current build with kernels compiled for this exact GPU generation, and explicitly configured settings instead of shipped defaults. The convenience layer costs about a third of your single-user speed on this model. Now it's a measured number instead of a rumour.
An 18.6GB model on a 16GB card means something has to give, and each engine gives differently.
| Qwen3-Coder-30B | TTFT | Prefill (long) | Decode | 8 clients |
|---|---|---|---|---|
| Ollama | 0.37 s | 1,164 tok/s | 67 tok/s | 60 tok/s |
| llama.cpp | 0.40 s | 1,100 tok/s | 71 tok/s | 139 tok/s |
| vLLM | 0.21 s | 2,580 tok/s | 59 tok/s | 179 tok/s |
llama.cpp wins the single-user case again, and the way it does it is genuinely clever. Qwen3-Coder is a mixture-of-experts model. Every layer has 128 small expert networks, and each token only wakes up 8 of them. llama.cpp has a flag built for exactly this shape, --n-cpu-moe. Instead of streaming weights to the GPU, it sends the work to the weights: the expert layers live in system RAM and the CPU computes them right there, while the bandwidth-hungry part, attention and the cache, stays on the GPU. Since each token only wakes a few experts, the CPU barely works. An 18GB model doing 71 tokens a second on a 16GB card is, honestly, amazing.
Ollama handled the oversized model without being asked, it auto-splits between GPU and CPU. 67 tokens a second, perfectly usable. vLLM's generic offload managed 59 single-stream but again won time-to-first-token, prefill, and the 8-client column. A hint of things to come.
Gemma 4 12B leaves 6 or 7GB of headroom. For the first time, vLLM loaded natively: no offload, no compromises, all its machinery switched on.
| Gemma 4 12B | TTFT | Prefill (long) | Decode | 8 clients |
|---|---|---|---|---|
| Ollama | 0.33 s | 2,203 tok/s | 87 tok/s | 78 tok/s |
| llama.cpp | 0.13 s | 2,584 tok/s | 91 tok/s | 232 tok/s |
| vLLM | 0.05 s | 3,366 tok/s | 81 tok/s | 419 tok/s |
Single user, the three are close: 91, 87, 81. You'd struggle to feel the difference in a chat window. But look at the 8-client column. vLLM served 419 tokens a second, nearly double llama.cpp and more than five times Ollama. The same burst of 8 requests took vLLM 4.9 seconds, llama.cpp 8.8, and Ollama 26.3. Sit with that spread for a second: same GPU, same model, and one engine returns the whole batch five times sooner than another.
That's what vLLM is for. Its poor showing in round one wasn't the engine, it was the offload. Give it resident weights and parallel traffic, and you can see exactly why datacenters run it.
Across all three rounds, one pattern repeated: 8 clients got the same total throughput as 1. Out of the box, Ollama serves requests one at a time. It's not broken, it's a default, and it's fixable with a single environment variable, OLLAMA_NUM_PARALLEL. But I suspect almost nobody sets it, which means almost every multi-user Ollama box out there is quietly serving people from a queue. If your Ollama serves more than one person, or one person who fires parallel requests from scripts, set it today. It's the cheapest performance win in this whole post.
That's the wrong question, and getting the wrong question out of the way was the real result of these evenings. The right question is: does your model fit, and how many people are hitting it?
On my own box, llama.cpp is now the serving engine behind everything single-user, Ollama stays as the quick way to pull and try models, and vLLM earned the job I originally doubted it could do here: batch work. My model eval suite fires eight requests at a time, which is exactly the traffic vLLM eats for breakfast.
Same machine throughout: RTX 5070 Ti 16GB, Ryzen 7 9700X, 128GB DDR5, one engine on the GPU at a time, temperature 0, streamed responses, every prompt uniquely tagged so prefix caching couldn't cheat on prefill. Cross-engine model files aren't byte-identical where noted: the Qwen round compares Q4_K_M against AWQ, and vLLM's Gemma checkpoint is Google's official quantized release at 10.3GB against the 7.4GB file the others served, which explains part of its single-stream gap, decode speed follows bytes. vLLM ran handicapped in rounds one and two (eager mode, short context, forced by the memory squeeze) and unhandicapped in round three. And a headless box would hand every engine back my desktop's 1GB, which would likely have turned round one's "can't load" into "loads, barely". If you want the deeper writeup on how GGUF, expert offload, and paged attention actually work under the hood, tell me, that's a post of its own.
First published on ringarc.ai on 7 August 2026. Comments and corrections: vikas@probearc.ai.