Multi GPU Training with Unsloth
📜 Get LIFETIME repo access at https://Trelis.com/ADVANCED-fine-tuning
🗝️ Get Trelis Multi-Repo Bundle (https://Trelis.com/multi-repo-bundle)
Access all SEVEN Trelis Github Repos (-robotics, -vision, -evals, -fine-tuning, -inference, -voice, -time-series)
Support via Github Issues & Trelis’ Private Discord
Tip: If you subscribe here on YouTube, click the bell to be notified of new vids
🛠️ (NEW) Trelis Fine-tuning Workshops: https://trelis.com/workshops-and-semi...
💡 Need Technical or Market Assistance?
Book a Consult Here: https://forms.gle/wJXVZXwioKMktjyVA
🤝 Are You a Top Developer?
Work for Trelis: https://trelis.com/jobs/
💸 Starting a New Project/Venture?
Apply for a Trelis Grant: https://trelis.com/trelis-ai-grants/
📧 Get Trelis AI Tutorials by Email
Subscribe on Substack: https://trelis.substack.com
Video Links:
Slides: https://docs.google.com/presentation/...
Unsloth multi-gpu issue: https://github.com/unslothai/unsloth/...
TIMESTAMPS:
0:00 Faster training with multiple GPUs
0:39 Video Overview
1:24 Data parallel versus Pipeline Parallel versus Fully Sharded Data Parallel
6:38 Downloading a jupyter notebook as a python script for multi-gpu, e.g. an unsloth notebook
7:44 Unsloth vs Transformers for multi-gpu
8:13 Modifying a fine-tuning script for distributed data parallel
9:03 Starting up a GPU in one-click for fine-tuning
10:27 Converting a jupyter notebook to a python script
11:30 Installation notes for unsloth and tensorboard, and uv
13:32 Script modifications required for DDP
18:50 Training script run-through, for LoRA
22:46 Setting gradient accumulation steps
24:07 Dataset loading
26:22 Setting up the run name and training parameters
29:30 Running without multi-gpu (single gpu check)
35:47 Running with multiple GPUs using accelerate config (btw torch run can result in run hangs)
41:02 Sanity check of running with accelerate and a single gpu
44:48 Open (at time of recording) issues with loss reporting and using unsloth with batch size larger than one
53:11 Conclusion and shout-outs to spr1nter and rakshith