Why AIs Need 8-Bit Numbers (It makes perfect sense!)
8-bit numbers are what old video games used, right? Sure, but AI needs them, too. This video dispels some misconceptions about 8-bit numbers and then explains why some AIs use 8-bit numbers rather than more precise 32-bit or 64-bit numbers. It also touches on an answer to the question, What is AI? In addition, the video covers basic concepts about AI, such as what training and inference are, how fast AI is growing, and why the size of AI models and the speed of current AI growth present engineers with considerable difficulty.
Sorry about the audio quality. I'm hoping to buy a new mic soon!
GREAT resource, survey on quantization in NNs, also contains info on quantization below 8-bits:
https://arxiv.org/pdf/2103.13630.pdf
FP8 white paper from NVidia, ARM, and Intel:
https://arxiv.org/pdf/2209.05433.pdf
More info on FP8, with examples:
https://docs.nvidia.com/deeplearning/...
Use of FP8 and Int8/good general summary article:
https://www.hpcwire.com/2023/08/07/th...
What is quantization (general-audience article/not a journal):
https://www.allaboutcircuits.com/tech...
More detailed info on int8 quantization in Pytorch:
https://developer.nvidia.com/blog/ach...
Both size and movement of data are problems, as well as summary of hardware solutions to this, also graph of parameter growth:
https://ieeexplore.ieee.org/document/...
Memory is a problem, especially memory bandwidth:
https://dl.acm.org/doi/abs/10.1145/34...
https://arxiv.org/pdf/2303.12310.pdf
Speed of int8 in A100:
https://www.nvidia.com/en-us/data-cen...
Hardware solutions for memory bandwidth (article):
https://www.embedded.com/overcoming-t...
NVidia A100 data sheet, also containing info on int8 speed:
https://www.nvidia.com/content/dam/en...
NVidia H100 data sheet:
https://resources.nvidia.com/en-us-te...
Graph of energy usage for different data representations:
https://arxiv.org/pdf/2103.13630.pdf
Edge AI use cases:
https://blogs.nvidia.com/blog/what-is...
Quantization at 8-bits maintains performance (int8 in first link, FP8 in second):
https://arxiv.org/pdf/2206.02915.pdf
https://arxiv.org/pdf/2004.09602.pdf
Further resources on quantization below 8 bits:
https://arxiv.org/pdf/2103.13630.pdf
https://arxiv.org/pdf/2302.04304.pdf
4-bit quantization of diffusion models:
https://arxiv.org/pdf/2302.04304.pdf