Fine tuning Optimizations - DoRA, NEFT, LoRA+, Unsloth
➡️ ADVANCED-fine-tuning Repo: https://trelis.com/advanced-fine-tuni...
➡️ ADVANCED-inference Repo: https://trelis.com/enterprise-server-...
➡️ Trelis Function-calling Models: https://trelis.com/function-calling/
➡️ One-click Fine-tuning & Inference Templates: https://github.com/TrelisResearch/one...
➡️ Trelis Newsletter: https://Trelis.Substack.com
➡️ Tip Jar and Discord: https://ko-fi.com/trelisresearch
VIDEO RESOURCES:
-Slides: https://docs.google.com/presentation/...
-DoRA: https://arxiv.org/abs/2402.09353
-LoRA+: https://arxiv.org/abs/2402.12354
-Unsloth: https://github.com/unslothai/unsloth
-NEFT: https://arxiv.org/abs/2310.05914
-Transformers integration: https://huggingface.co/docs/trl/main/...
TIMESTAMPS:
0:00 Improving on LoRA
0:30 Video Overview
1:25 How does LoRA work?
4:26 Understanding DoRA
8:42 NEFT - Adding Noise to Embeddings
7:12 LoRA Plus
9:34 Unsloth for fine-tuning speedups
11:27 Comparing LoRA+, Unsloth, DoRA, NEFT
13:02 Notebook Setup and LoRA
23:17 DoRA Notebook Walk-through
25:40 NEFT Notebook Example
26:45 LoRA Plus
30:10 Unsloth
31:18 Final Recommendation