Quantizing LLMs - How & Why (8-Bit, 4-Bit, GGUF & More)
Quantizing models for maximum efficiency gains!
Resources:
Model Quantized: https://huggingface.co/AdamLucek/Orpo...
Quantization Colab Notebook: https://colab.research.google.com/dri...
HF 8-bit Blog: https://huggingface.co/blog/hf-bitsan...
HF 4-Bit Blog: https://huggingface.co/blog/4bit-tran...
GGUF Overview: https://huggingface.co/docs/hub/gguf
Llama.cpp: https://github.com/ggerganov/llama.cp...
GGUF Model Made in Video: https://huggingface.co/AdamLucek/Orpo...
Maxime Labonne Quantization Blog: https://mlabonne.github.io/blog/posts...
Chapters:
00:00 - What Is Quantization?
02:19 - How Are Weights Stored?
03:22 - What is Binary?
06:26 - What are Floating Point Numbers?
10:38 - What Data Types are Used for LLMs?
12:02 - Does Quantization Negatively Affect LLMs?
15:08 - Code: Quantizing with BitsAndBytes
17:34 - Code: Comparing Quantized Layers
18:36 - Code: Comparing Text Generation
21:57 - Code: GGUF Quantization Overview
23:41 - Code: Quantizing with Llama.cpp
25:44 - Final Thoughts on Quantization
#ai #coding #deeplearning