Build an LLM from Scratch 5: Pretraining on Unlabeled Data @SebastianRaschka

Uploaded March 2025 | Updated September 2026, 2 weeks ago

Links to the book:
- amzn.to/4fqvn0D (Amazon)
- https://mng.bz/M96o (Manning)

Link to the GitHub repository: github.com/rasbt/LLMs-from-scratch

This video explains how to pretrain a LLM from scratch.

00:00 5.1.1 Using GPT to generate text
17:23 5.1.2 Calculating the text generation loss: cross-entropy and perplexity
44:43 5.1.3 Calculating the training and validation set losses
1:10:27 5.2 Training an LLM
1:36:36 5.3 Decoding strategies to control randomness
1:39:51 5.3.1 Temperature scaling
1:53:24 5.3.2 Top-k sampling
2:01:40 5.3.3 Modifying the text generation function
2:12:24 5.4 Loading and saving model weights in PyTorch
2:16:44 5.5 Loading pretrained weights from OpenAI

You can find additional bonus materials on GitHub:

Pretraining GPT on the Project Gutenberg Dataset, github.com/rasbt/LLMs-from-scratch/tree/main/ch05/03_bonus_pretraining_on_gutenberg

PyTorch Performance Tips for Faster LLM Training, github.com/rasbt/LLMs-from-scratch/tree/main/ch05/10_llm-training-speed

Converting the GPT-2 architecture into Llama 2 and Llama 3, github.com/rasbt/LLMs-from-scratch/tree/main/ch05/07_gpt_to_llama

Build an LLM from Scratch 5: Pretraining on Unlabeled Data

L19.5.2.3 BERT: Bidirectional Encoder Representations from Transformers

13.3.1 L1-regularized Logistic Regression as Embedded Feature Selection (L13: Feature Selection)

L18.5: Tips and Tricks to Make GANs Work

L8.8 Softmax Regression Derivatives for Gradient Descent

L17.5 A Variational Autoencoder for Handwritten Digits in PyTorch Code Example

L12.5 Choosing Different Optimizers in PyTorch

L9.3.3 Multilayer Perceptron in PyTorch Code Example Part 3/3 (Script Setup)

L18.4: A GAN for Generating Handwritten Digits in PyTorch Code Example