Uploaded November 2024 | Updated September 2026, 2 weeks ago
In this deep dive video, we zoom in on two popular techniques for parameter-efficient training, LoRA/QLoRA and Spectrum. We discuss their mathematical foundations in detail, including Singular Value Decomposition (SVD). Then, we look at some benchmarks on popular Small Language Models, Mistral-7b and Llama-3.1-8b. We conclude that Spectrum is the better choice, both in terms of training speed and model quality, and is even competitive with the accuracy of full fine-tuning.
⭐️⭐️⭐️ Don't forget to subscribe to be notified of future videos. You can become a channel member and enjoy exclusive perks: details at youtube.com/channel/UCVonoXm3SI_Q0ZNHd5JPawA/join
You can also follow me on Medium at julsimon.medium.com or Substack at https://julsimon.substack.com. ⭐️⭐️⭐️
* Slides: fr.slideshare.net/slideshow/deep-dive-parameter-efficient-model-adaptation-with-lora-and-spectrum/273716516
* Spectrum: github.com/cognitivecomputations/spectrum
* Axolotl: github.com/axolotl-ai-cloud/axolotl
* lm_eval: github.com/EleutherAI/lm-evaluation-harness
* Configuration files: gitlab.com/juliensimon/arcee-demos/-/tree/main/spectrum?ref_type=heads
00:00 Introduction
01:12 A typical model adaptation workflow
04:10 Challenges of model adaptation
10:35 Singular Value Decomposition
14:20 Low-rank approximation with SVD
17:25 Running an SVD example in Python
22:09 LoRA
26:35 QLoRA
27:25 Challenges with LoRA
32:01 Spectrum
40:25 Using Spectrum
42:00 Mistral-7b benchmarks
43:35 Llama-3-8b benchmarks
44:30 GPU RAM usage and training time
45:45 My benchmark on Arcee SuperNova Lite
49:45 Conclusion
In this deep dive video, we zoom in on two popular techniques for parameter-efficient training, LoRA/QLoRA and Spectrum. We discuss their mathematical foundations in detail, including Singular Value Decomposition (SVD). Then, we look at some benchmarks on popular Small Language Models, Mistral-7b and Llama-3.1-8b. We conclude that Spectrum is the better choice, both in terms of training speed and model quality, and is even competitive with the accuracy of full fine-tuning.
⭐️⭐️⭐️ Don't forget to subscribe to be notified of future videos. You can become a channel member and enjoy exclusive perks: details at youtube.com/channel/UCVonoXm3SI_Q0ZNHd5JPawA/join
You can also follow me on Medium at julsimon.medium.com or Substack at https://julsimon.substack.com. ⭐️⭐️⭐️
* Slides: fr.slideshare.net/slideshow/deep-dive-parameter-efficient-model-adaptation-with-lora-and-spectrum/273716516
* Spectrum: github.com/cognitivecomputations/spectrum
* Axolotl: github.com/axolotl-ai-cloud/axolotl
* lm_eval: github.com/EleutherAI/lm-evaluation-harness
* Configuration files: gitlab.com/juliensimon/arcee-demos/-/tree/main/spectrum?ref_type=heads
00:00 Introduction
01:12 A typical model adaptation workflow
04:10 Challenges of model adaptation
10:35 Singular Value Decomposition
14:20 Low-rank approximation with SVD
17:25 Running an SVD example in Python
22:09 LoRA
26:35 QLoRA
27:25 Challenges with LoRA
32:01 Spectrum
40:25 Using Spectrum
42:00 Mistral-7b benchmarks
43:35 Llama-3-8b benchmarks
44:30 GPU RAM usage and training time
45:45 My benchmark on Arcee SuperNova Lite
49:45 Conclusion
