Uploaded December 2023 | Updated September 2026, 2 weeks ago
Sebastian's books: sebastianraschka.com/books
Links:
- LoRA: Low-Rank Adaptation of Large Language Models, arxiv.org/abs/2106.09685
- LitGPT: github.com/Lightning-AI/lit-gpt
- LitGPT LoRA Tutorial: github.com/Lightning-AI/lit-gpt/blob/main/tutorials/finetune_lora.md
Low-rank adaptation (LoRA) stands as one of the most popular and effective methods for efficiently training custom Large Language Models (LLMs). As practitioners of open-source LLMs, we regard LoRA as a crucial technique in our toolkit.
In this talk, I will delve into some practical insights gained from running hundreds of experiments with LoRA, addressing questions such as: How much can I save with quantized LoRA? Are Adam optimizers memory-intensive? Should we train for multiple epochs? How do we choose the LoRA rank?
---
To support this channel, please consider purchasing a copy of my books: sebastianraschka.com/books
---
twitter.com/rasbt
linkedin.com/in/sebastianraschka
magazine.sebastianraschka.com
Sebastian's books: sebastianraschka.com/books
Links:
- LoRA: Low-Rank Adaptation of Large Language Models, arxiv.org/abs/2106.09685
- LitGPT: github.com/Lightning-AI/lit-gpt
- LitGPT LoRA Tutorial: github.com/Lightning-AI/lit-gpt/blob/main/tutorials/finetune_lora.md
Low-rank adaptation (LoRA) stands as one of the most popular and effective methods for efficiently training custom Large Language Models (LLMs). As practitioners of open-source LLMs, we regard LoRA as a crucial technique in our toolkit.
In this talk, I will delve into some practical insights gained from running hundreds of experiments with LoRA, addressing questions such as: How much can I save with quantized LoRA? Are Adam optimizers memory-intensive? Should we train for multiple epochs? How do we choose the LoRA rank?
---
To support this channel, please consider purchasing a copy of my books: sebastianraschka.com/books
---
twitter.com/rasbt
linkedin.com/in/sebastianraschka
magazine.sebastianraschka.com
