Compress Deep Learning models 10,000x with Probabilistic Hash Functions
Neural Scaling Law informally states that an increase in model size and data automatically improves AI. However, we have reached a point where the growth has reached a tipping end where the cost and energy associated with AI are becoming prohibitive.
This talk will demonstrate the algorithmic progress that can exponentially reduce the compute and memory cost of training and inference with neural networks. We will show how data structures,
particularly randomized hash tables, can be used to design an efficient "associative memory" that reduces the number of multiplications associated with the training of the neural networks.
Implementation of this algorithm challenges the common knowledge prevailing in the community that specialized processors like GPUs are significantly superior to CPUs for training large neural networks. The resulting algorithm is orders of magnitude cheaper and energy-efficient. Our careful implementations can train billions of parameter recommendations and NLP models on commodity desktop CPUs significantly faster than top-of-the-line TensorFlow alternatives on the most potent A100 GPU clusters, with the same or better accuracies. The same idea can also result in more than 50x faster and cheaper inference.
In the end, I will highlight a cache-friendly compression scheme that can compress embedding models by 10000x (100GB Embedding Table to 10MB) and still achieves the MLPerf benchmark AUC of 0.8025 on the Terabyte click-through Criteo data, getting significant inference speedup for free.
Bio: Anshumali Shrivastava is an associate professor in the computer science department at Rice University. He is also the Founder and CEO of ThirdAI Corp, a company that is democratizing AI to commodity hardware through software innovations. His broad research interests include probabilistic algorithms for resource-frugal deep learning. In 2018, Science news named him one of the Top-10 scientists under 40 to watch. He is a recipient of the National Science Foundation CAREER Award, a Young Investigator Award from the Air Force Office of Scientific Research, a machine learning research award from Amazon, and a Data Science Research Award from Adobe. He has won numerous paper awards, including Best Paper Award at NIPS 2014, MLSys 2022, and Most Reproducible Paper Award at SIGMOD 2019. His work on efficient machine learning technologies on CPUs has been covered by popular press including Wall Street Journal, New York Times, TechCrunch, NDTV, Engadget, Ars technica, etc.
https://www.thirdai.com/
https://www.cs.rice.edu/~as143/
/ shrivastava-anshumali-58098b6
0:00 Chapter Intro
2:38 Talk Intro
4:05 Presentation
4:54 We need Large Models
8:27 Simple Architectures are Fine: Cannot Ignore Size
10:38 What's Next: A Universal AI
13:18 Single API for all ML problems
14:15 ThirdAI's Universal AI
15:08 Universal AI seems ideal, but…how close are we?
16:28 Training Time and Cost Drives AI Innovation: It is all Model Search
19:44 The breakthrough discovery
20:42 Current practice is not sustainable: Hitting a Wall
22:00 Lets simply go around the wall
22:31 Dense Cmoputation is wasteful for Large Models
25:50 Hope for Future: Dynamic Sparsity
28:13 Good Old Data Structures to Rescue
30:19 Finding Influential Neurons/Parameters
31:02 Really! Can query be faster than a matrix multiplication?
31:41 Algorithms and Data structures to Rescue. Sparsity via Hash Tables
35:47 Data Parallel (near) Asynchronous Gradient Descent
45:05 Compressing Embedding Tables
1:04:12 Most Exciting Time to Revisit Algorithms and Data Structures for Deep Learning Systems
1:06:13 Acknowledgements