Multi-GPU Just Got 3-4x Faster

For a long time, the fast way to run a model across several GPUs lived inside heavyweight engines like vLLM and TensorRT-LLM, and mostly on NVIDIA. If you ran llama.cpp instead, your second and third cards basically took turns. As of April, that is over. A change merged into llama.cpp makes every card work on every token, and it runs the same way on NVIDIA, AMD, or Vulkan.

The Old Way: Cards Taking Turns

llama.cpp could always see more than one GPU. It just used them politely. Card one held the first half of the model, card two held the rest, and they passed the work back and forth: while one computed, the others waited. Multi-GPU on the spec sheet, single-GPU speed in real life.

The technique that actually keeps every card busy, tensor parallelism, splits the math inside each layer. That used to be the property of the big inference engines: vLLM, TensorRT-LLM, SGLang. Powerful, production-grade, and a project to stand up. TensorRT-LLM does not even leave the NVIDIA lot.

What Changed

llama.cpp learned to do tensor parallelism itself. A new meta-backend treats the GPUs in your machine as one logical card and splits each layer's work across all of them at once, so they compute the same token together instead of trading it hand to hand. The result is three to four times faster than the take-turns method, with every card pinned at 100 percent.

The part that matters beyond speed: the logic sits above any one vendor's libraries. It runs on CUDA, on AMD through ROCm, and on Vulkan. Your hardware, not a permission slip from Santa Clara. Apple's Metal is not in yet; it is on the list.

When It's Worth the Hassle

Not everyone needs this. One card, model fits, you are done. Nothing here changes your day.

It is for the moment you hit the single-card memory wall. A 70B that will not load on one 24GB card now spreads across two or three and keeps generating at a usable clip, with the data never leaving machines you control. And you can build that rig out of cheaper AMD cards instead of paying the going rate for enterprise NVIDIA.

The Catch

It is new, and it shows. The ROCm and Vulkan paths are slower and shakier than CUDA today, and setup is more than a one-line install. If you are running a rack of NVIDIA cards as a real production server, vLLM still does that job better. llama.cpp's home turf is the machine on your desk, and this change makes that machine punch well above its weight.

The fast lane used to have a toll and a brand on the gate. Now your GPUs, whatever is in them, just run. Build the rig. Load the model. Watch every card light up.