Run SLMs locally: Llama.cpp vs. MLX with 10B and 32B Arcee models
In this video, we run local inference on an Apple M3 MacBook with llama.cpp and MLX, two projects that optimize and accelerate small language models on CPU platforms. For this purpose, we use two new Arcee open-source models distilled from DeepSeek-v3: Virtuoso Lite 10B and Virtuoso Medium v2 32B.
First, we download the two models from the Hugging Face hub with the Hugging Face CLI. Then, we go through the step-by-step installation procedure for llama.cpp and MLX. Next, we optimize and quantize the models to 4-bit precision for maximum acceleration. Finally, we run inference and look at performance numbers. So, who's fastest? Watch and find out :)
If you’d like to understand how Arcee AI can help your organization build scalable and cost-efficient AI solutions, don't hesitate to contact [email protected] or book a demo at https://www.arcee.ai/book-a-demo.
⭐️⭐️⭐️ Don't forget to subscribe to be notified of future videos. You can also follow me on Medium at / julsimon or Substack at https://julsimon.substack.com. ⭐️⭐️⭐️
Blog post: https://www.arcee.ai/blog/virtuoso-li...
Virtuoso Lite: https://huggingface.co/arcee-ai/Virtu...
Virtuoso Medium v2: https://huggingface.co/arcee-ai/Virtu...
Llama.cpp: https://github.com/ggerganov/llama.cpp
MLX for language models: https://github.com/ml-explore/mlx-exa...
00:00 Introduction
00:40 A quick look at Virtuoso-Lite and Virtuoso-Medium-v2
03:10 Downloading the models from Hugging Face
04:30 Building llama.cpp from source
06:15 Converting Hugging Face models to GGUF
09:20 Quantizing models with llama.cpp
11:20 Running local inference with llama.cpp
15:00 Installing MLX
11:20 Running local inference with MLX
19:45 Conclusion