How Do We Get MASSIVE Model To Run On Device? Quantization Explained.
Every time I do a video about a model I get a comment saying "Well you never said what it takes to run it!" Well since I am not inside of your computer the answer is - "it depends".
In this video I want to take a simple explainer or crash course on Quantization for LLMs. That is those Q_ files you see on HuggingFace. Where do they come from? How are they made? What is really going on with this process.
This is a brief explainer video about this whole process so that the next time a model drops - you can looks at these quants with confidence and ultimately know what it might take to run them.
Links
AnythingLLM: https://anythingllm.com
About GGUF: https://github.com/ggml-org/ggml/blob...
Example Config model: https://huggingface.co/Qwen/Qwen3.5-2...
Chapters
0:00 What even are these files?
1:30 What is GGUF?
2:32 What is a quant (or Quantization)
7:08 Who is making these files?
8:40 Looking at a real example
9:39 All about Quants and How They Work]
18:00 About Perplexity
19:39 Choosing the Right Quant
20:39 How memory is used
21:38 Calculate Memory for Model
22:51 Calculate the KV Cache size (Context window memory)
26:19 Crash Course is over