Spectrum: Training Domain-Adapted SLMs
We explore the growing trend of Small Language Models (SLMs) and their specialization in domain-specific tasks. Discover how SLMs, paired with advanced fine-tuning techniques like Spectrum, are revolutionizing AI. Spectrum leverages a signal-to-noise ratio (SNR) to optimize which layers of a model to fine-tune, balancing performance and cost effectively. We'll dive into the details of Spectrum, comparing it to industry standards like Low-Rank Adaptation (LoRA) and Quantized LoRA (QLoRA). Learn from experts at Arcee.ai, the minds behind the Spectrum paper, and uncover the practical applications of their innovative methods. If you're aiming to enhance your domain-adapted language models with cutting-edge techniques, this event is a must-attend.
Join us every Wednesday at 1pm EST for our live events. SUBSCRIBE NOW to get notified!
Speakers:
Dr. Greg, Co-Founder & CEO AI Makerspace
/ gregloughane
The Wiz, Co-Founder & CTO AI Makerspace
/ csalexiuk
Apply for The AI Engineering Bootcamp on Maven today!
https://bit.ly/AIEbootcamp
LLM Foundations - Email-based course
https://aimakerspace.io/llm-foundations/
For team leaders, check out!
https://aimakerspace.io/gen-ai-upskil...
Join our community to start building, shipping, and sharing with us today!
/ discord
How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/z96cKbg3epXXqwtG6
00:00:00 Introduction to Spectrum: A New Fine-Tuning Method
00:04:03 Introduction to Spectrum and Key Team
00:07:49 Learning Efficiency and Forgetting Less in AI Models
00:11:26 Optimizing Multi-GPU Training in AI Development
00:15:24 Emergence of Spectrum in AI Models
00:19:17 Evolution of the AI Space and Collaborations
00:22:45 Understanding Randomized Matrix Theory and Marchenko-Pastur Distribution
00:27:00 Understanding Noise and Signal with Marenco-Pasture Analysis
00:31:28 Understanding Eigenvalues and Matrix Redundancy
00:35:15 Advanced Pre-training Techniques in AI
00:39:02 Integrating SNR in Fine-Tuning Language Models
00:42:45 Introduction to Spectrum and JSON File Generation
00:46:07 Utilizing Axel and Spectrum for Model Fine-tuning
00:49:19 Understanding Google's Software Development Kit Updates
00:52:47 Leveraging Spectrum and Cura for Efficient Model Training
00:56:29 Advancements in Model Training: Spectrum vs. Laser RMT
01:00:04 Farewell and Weekly Wrap-Up