CPU Inference: Myth or Reality? | Qwen3-Coder 480B, Quantization and CPU+GPU Hybrid
Link to the commands that were used: https://github.com/pavtiger/articles/...
In this video, ServerFlow specialists discuss the topic of LLM inference on the CPU. We will try to delve into the issue and understand in detail when running models on the processor really makes sense. The video covers the popular Ollama and llama.cpp engines. For testing, we use a server with two AMD EPYC 7542 processors and 2 TB of RAM. The test model is the popular Qwen3-Coder 480B with different quantization levels.
Paradoxically, experiments have shown that using two processors reduces the speed of LLM inference due to the need to exchange data via the Infinity Fabric bus. The video includes visual graphs that show the difference in performance when running on one and two CPUs. Special attention is paid to comparing the inference speed in Ollama and llama.cpp. We will clearly show how much llama.cpp outperforms its competitor in terms of performance.
An interesting feature was the use of a hybrid CPU+GPU approach, when the MoE model "switch" runs on the GPU, and the "experts" calculations are performed on the CPU. This method allowed us to speed up the inference by about 30%. In the final, we will test different quantization levels - from extreme Q3_K_M to native launch of the model in BF16 format.
0:00 Introduction
0:56 Server configuration
2:06 Ollama
4:42 llama.cpp
7:08 GPU acceleration
8:01 Open-webui
8:26 Benchmarks
10:58 Conclusion
Our website - https://serverflow.ru/
ServerFlow blog - https://serverflow.ru/blog/