Lets build the GPT Tokenizer Andrej Karpathy 2024-02-20 | The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement two fundamental functions: encode() from strings to tokens, and decode() back from tokens to strings. In this lecture we build from scratch the Tokenizer used in the GPT series from OpenAI. In the process, we will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We'll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.Chapters:00:00:00 intro: Tokenization, GPT-2 paper, tokenization-related issues00:05:50 tokenization by example in a Web UI (tiktokenizer)00:14:56 strings in Python, Unicode code points00:18:15 Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-3200:22:47 daydreaming: deleting tokenization00:23:50 Byte Pair Encoding (BPE) algorithm walkthrough00:27:02 starting the implementation00:28:35 counting consecutive pairs, finding most common pair00:30:36 merging the most common pair00:34:58 training the tokenizer: adding the while loop, compression ratio00:39:20 tokenizer/LLM diagram: it is a completely separate stage00:42:47 decoding tokens to strings00:48:21 encoding strings to tokens00:57:36 regex patterns to force splits across categories01:11:38 tiktoken library intro, differences between GPT-2/GPT-4 regex01:14:59 GPT-2 encoder.py released by OpenAI walkthrough01:18:26 special tokens, tiktoken handling of, GPT-2/GPT-4 differences01:25:28 minbpe exercise time! write your own GPT-4 tokenizer01:28:42 sentencepiece library intro, used to train Llama 2 vocabulary01:43:27 how to set vocabulary set? revisiting gpt.py transformer01:48:11 training new tokens, example of prompt compression01:49:58 multimodal [image, video, audio] tokenization with vector quantization01:51:41 revisiting and explaining the quirks of LLM tokenization02:10:20 final recommendations02:12:50 ??? :)Exercises:- Advised flow: reference this document and try to implement the steps before I give away the partial solutions in the video. The full solutions if you're getting stuck are in the minbpe code github.com/karpathy/minbpe/blob/master/exercise.mdLinks:- Google colab for the video: colab.research.google.com/drive/1y0KnCFZvGVf_odSfcNAws6kcDD7HsI0L?usp=sharing- GitHub repo for the video: minBPE github.com/karpathy/minbpe- Playlist of the whole Zero to Hero series so far: youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ- our Discord channel: discord.gg/3zy8kqD9Cp- my Twitter: twitter.com/karpathySupplementary links:- tiktokenizer https://tiktokenizer.vercel.app- tiktoken from OpenAI: github.com/openai/tiktoken- sentencepiece from Google github.com/google/sentencepiece
Lets reproduce GPT-2 (124M) Andrej Karpathy 2024-06-09 | We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training run following the GPT-2 and GPT-3 paper and their hyperparameters, then we hit run, and come back the next morning to see our results, and enjoy some amusing model generations. Keep in mind that in some places this video builds on the knowledge from earlier videos in the Zero to Hero Playlist (see my channel). You could also see this video as building my nanoGPT repo, which by the end is about 90% similar.Links:- build-nanogpt GitHub repo, with all the changes in this video as individual commits: github.com/karpathy/build-nanogpt- nanoGPT repo: github.com/karpathy/nanoGPT- llm.c repo: github.com/karpathy/llm.c- my website: karpathy.ai- my twitter: twitter.com/karpathy- our Discord channel: discord.gg/3zy8kqD9CpSupplementary links:- Attention is All You Need paper: arxiv.org/abs/1706.03762- OpenAI GPT-3 paper: arxiv.org/abs/2005.14165 - OpenAI GPT-2 paper: d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf- The GPU I'm training the model on is from Lambda GPU Cloud, I think the best and easiest way to spin up an on-demand GPU instance in the cloud that you can ssh to: lambdalabs.com Chapters:00:00:00 intro: Let’s reproduce GPT-2 (124M)00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint00:13:47 SECTION 1: implementing the GPT-2 nn.Module00:28:08 loading the huggingface/GPT-2 parameters00:31:00 implementing the forward pass to get logits00:33:31 sampling init, prefix tokens, tokenization00:37:02 sampling loop00:41:47 sample, auto-detect the device00:45:50 let’s train: data batches (B,T) → logits (B,T,C)00:52:53 cross entropy loss00:56:42 optimization loop: overfit a single batch01:02:00 data loader lite01:06:14 parameter sharing wte and lm_head01:13:47 model initialization: std 0.02, residual init01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms01:39:38 float16, gradient scalers, bfloat16, 300ms01:48:15 torch.compile, Python overhead, kernel fusion, 130ms02:00:18 flash attention, 96ms02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping02:21:06 learning rate scheduler: warmup + cosine decay02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms02:34:09 gradient accumulation02:46:52 distributed data parallel (DDP)03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU)03:23:10 validation data split, validation loss, sampling revive03:28:23 evaluation: HellaSwag, starting the run03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA03:59:39 summary, phew, build-nanogpt github repoCorrections:I will post all errata and followups to the build-nanogpt GitHub repo (link above)SuperThanks:I experimentally enabled them on my channel yesterday. Totally optional and only use if rich. All revenue goes to to supporting my work in AI + Education.
[1hr Talk] Intro to Large Language Models Andrej Karpathy 2023-11-23 | This is a 1 hour general-audience introduction to Large Language Models: the core technical component behind systems like ChatGPT, Claude, and Bard. What they are, where they are headed, comparisons and analogies to present-day operating systems, and some of the security-related challenges of this new computing paradigm.As of November 2023 (this field moves fast!).Context: This video is based on the slides of a talk I gave recently at the AI Security Summit. The talk was not recorded but a lot of people came to me after and told me they liked it. Seeing as I had already put in one long weekend of work to make the slides, I decided to just tune them a bit, record this round 2 of the talk and upload it here on YouTube. Pardon the random background, that's my hotel room during the thanksgiving break.- Slides as PDF: drive.google.com/file/d/1pxx_ZI7O-Nwl7ZLNk5hI3WzAsTLwvNU7/view?usp=share_link (42MB)- Slides. as Keynote: drive.google.com/file/d/1FPUpFMiCkMRKPFjhi9MAhby68MHVqe8u/view?usp=share_link (140MB)Few things I wish I said (I'll add items here as they come up):- The dreams and hallucinations do not get fixed with finetuning. Finetuning just "directs" the dreams into "helpful assistant dreams". Always be careful with what LLMs tell you, especially if they are telling you something from memory alone. That said, similar to a human, if the LLM used browsing or retrieval and the answer made its way into the "working memory" of its context window, you can trust the LLM a bit more to process that information into the final answer. But TLDR right now, do not trust what LLMs say or do. For example, in the tools section, I'd always recommend double-checking the math/code the LLM did.- How does the LLM use a tool like the browser? It emits special words, e.g. |BROWSER|. When the code "above" that is inferencing the LLM detects these words it captures the output that follows, sends it off to a tool, comes back with the result and continues the generation. How does the LLM know to emit these special words? Finetuning datasets teach it how and when to browse, by example. And/or the instructions for tool use can also be automatically placed in the context window (in the “system message”).- You might also enjoy my 2015 blog post "Unreasonable Effectiveness of Recurrent Neural Networks". The way we obtain base models today is pretty much identical on a high level, except the RNN is swapped for a Transformer. http://karpathy.github.io/2015/05/21/rnn-effectiveness- What is in the run.c file? A bit more full-featured 1000-line version hre: github.com/karpathy/llama2.c/blob/master/run.cChapters:Part 1: LLMs00:00:00 Intro: Large Language Model (LLM) talk00:00:20 LLM Inference00:04:17 LLM Training00:08:58 LLM dreams00:11:22 How do they work?00:14:14 Finetuning into an Assistant00:17:52 Summary so far00:21:05 Appendix: Comparisons, Labeling docs, RLHF, Synthetic data, LeaderboardPart 2: Future of LLMs00:25:43 LLM Scaling Laws00:27:43 Tool Use (Browser, Calculator, Interpreter, DALL-E)00:33:32 Multimodality (Vision, Audio)00:35:00 Thinking, System 1/200:38:02 Self-improvement, LLM AlphaGo00:40:45 LLM Customization, GPTs store00:42:15 LLM OSPart 3: LLM Security00:45:43 LLM Security Intro00:46:14 Jailbreaks00:51:30 Prompt Injection00:56:23 Data poisoning00:58:37 LLM Security conclusionsEnd00:59:23 Outro
Lets build GPT: from scratch, in code, spelled out. Andrej Karpathy 2023-01-17 | We build a Generatively Pretrained Transformer (GPT), following the paper "Attention is All You Need" and OpenAI's GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework and basics of tensors and PyTorch nn, which we take for granted in this video.Links:- Google colab for the video: colab.research.google.com/drive/1JMLa53HDuA-i7ZBmqV7ZnA3c_fvtXnx-?usp=sharing- GitHub repo for the video: github.com/karpathy/ng-video-lecture- Playlist of the whole Zero to Hero series so far: youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ- nanoGPT repo: github.com/karpathy/nanoGPT- my website: karpathy.ai- my twitter: twitter.com/karpathy- our Discord channel: discord.gg/3zy8kqD9CpSupplementary links:- Attention is All You Need paper: arxiv.org/abs/1706.03762- OpenAI GPT-3 paper: arxiv.org/abs/2005.14165 - OpenAI ChatGPT blog post: openai.com/blog/chatgpt- The GPU I'm training the model on is from Lambda GPU Cloud, I think the best and easiest way to spin up an on-demand GPU instance in the cloud that you can ssh to: lambdalabs.com . If you prefer to work in notebooks, I think the easiest path today is Google Colab.Suggested exercises:- EX1: The n-dimensional tensor mastery challenge: Combine the `Head` and `MultiHeadAttention` into one class that processes all the heads in parallel, treating the heads as another batch dimension (answer is in nanoGPT).- EX2: Train the GPT on your own dataset of choice! What other data could be fun to blabber on about? (A fun advanced suggestion if you like: train a GPT to do addition of two numbers, i.e. a+b=c. You may find it helpful to predict the digits of c in reverse order, as the typical addition algorithm (that you're hoping it learns) would proceed right to left too. You may want to modify the data loader to simply serve random problems and skip the generation of train.bin, val.bin. You may want to mask out the loss at the input positions of a+b that just specify the problem using y=-1 in the targets (see CrossEntropyLoss ignore_index). Does your Transformer learn to add? Once you have this, swole doge project: build a calculator clone in GPT, for all of +-*/. Not an easy problem. You may need Chain of Thought traces.)- EX3: Find a dataset that is very large, so large that you can't see a gap between train and val loss. Pretrain the transformer on this data, then initialize with that model and finetune it on tiny shakespeare with a smaller number of steps and lower learning rate. Can you obtain a lower validation loss by the use of pretraining?- EX4: Read some transformer papers and implement one additional feature or change that people seem to use. Does it improve the performance of your GPT?Chapters:00:00:00 intro: ChatGPT, Transformers, nanoGPT, Shakespearebaseline language modeling, code setup00:07:52 reading and exploring the data00:09:28 tokenization, train/val split00:14:27 data loader: batches of chunks of data00:22:11 simplest baseline: bigram language model, loss, generation00:34:53 training the bigram model00:38:00 port our code to a scriptBuilding the "self-attention"00:42:13 version 1: averaging past context with for loops, the weakest form of aggregation00:47:11 the trick in self-attention: matrix multiply as weighted aggregation00:51:54 version 2: using matrix multiply00:54:42 version 3: adding softmax00:58:26 minor code cleanup01:00:18 positional encoding01:02:00 THE CRUX OF THE VIDEO: version 4: self-attention01:11:38 note 1: attention as communication01:12:46 note 2: attention has no notion of space, operates over sets01:13:40 note 3: there is no communication across batch dimension01:14:14 note 4: encoder blocks vs. decoder blocks01:15:39 note 5: attention vs. self-attention vs. cross-attention01:16:56 note 6: "scaled" self-attention. why divide by sqrt(head_size)Building the Transformer01:19:11 inserting a single self-attention block to our network01:21:59 multi-headed self-attention01:24:25 feedforward layers of transformer block01:26:48 residual connections01:32:51 layernorm (and its relationship to our previous batchnorm)01:37:49 scaling up the model! creating a few variables. adding dropoutNotes on Transformer01:42:39 encoder vs. decoder vs. both (?) Transformers01:46:22 super quick walkthrough of nanoGPT, batched multi-headed self-attention01:48:53 back to ChatGPT, GPT-3, pretraining vs. finetuning, RLHF01:54:32 conclusionsCorrections: 00:57:00 Oops "tokens from the _future_ cannot communicate", not "past". Sorry! :)01:20:05 Oops I should be using the head_size for the normalization, not C
Building makemore Part 5: Building a WaveNet Andrej Karpathy 2022-11-21 | We take the 2-layer MLP from previous video and make it deeper with a tree-like structure, arriving at a convolutional neural network architecture similar to the WaveNet (2016) from DeepMind. In the WaveNet paper, the same hierarchical architecture is implemented more efficiently using causal dilated convolutions (not yet covered). Along the way we get a better sense of torch.nn and what it is and how it works under the hood, and what a typical deep learning development process looks like (a lot of reading of documentation, keeping track of multidimensional tensor shapes, moving between jupyter notebooks and repository code, ...).Links:- makemore on github: github.com/karpathy/makemore- jupyter notebook I built in this video: github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part5_cnn1.ipynb- collab notebook: colab.research.google.com/drive/1CXVEmCO_7r7WYZGb5qnjfyxTvQa13g5X?usp=sharing- my website: karpathy.ai- my twitter: twitter.com/karpathy- our Discord channel: discord.gg/3zy8kqD9CpSupplementary links:- WaveNet 2016 from DeepMind arxiv.org/abs/1609.03499- Bengio et al. 2003 MLP LM jmlr.org/papers/volume3/bengio03a/bengio03a.pdf Chapters:intro00:00:00 intro00:01:40 starter code walkthrough00:06:56 let’s fix the learning rate plot00:09:16 pytorchifying our code: layers, containers, torch.nn, fun bugsimplementing wavenet00:17:11 overview: WaveNet00:19:33 dataset bump the context size to 800:19:55 re-running baseline code on block_size 800:21:36 implementing WaveNet00:37:41 training the WaveNet: first pass00:38:50 fixing batchnorm1d bug00:45:21 re-training WaveNet with bug fix00:46:07 scaling up our WaveNetconclusions00:46:58 experimental harness00:47:44 WaveNet but with “dilated causal convolutions”00:51:34 torch.nn00:52:28 the development process of building deep neural nets00:54:17 going forward00:55:26 improve on my loss! how far can we improve a WaveNet on this data?
Building makemore Part 4: Becoming a Backprop Ninja Andrej Karpathy 2022-10-11 | We take the 2-layer MLP (with BatchNorm) from the previous video and backpropagate through it manually without using PyTorch autograd's loss.backward(): through the cross entropy loss, 2nd linear layer, tanh, batchnorm, 1st linear layer, and the embedding table. Along the way, we get a strong intuitive understanding about how gradients flow backwards through the compute graph and on the level of efficient Tensors, not just individual scalars like in micrograd. This helps build competence and intuition around how neural nets are optimized and sets you up to more confidently innovate on and debug modern neural networks.!!!!!!!!!!!!I recommend you work through the exercise yourself but work with it in tandem and whenever you are stuck unpause the video and see me give away the answer. This video is not super intended to be simply watched. The exercise is here:colab.research.google.com/drive/1WV2oi2fh9XXyldh02wupFQX0wh5ZC-z-?usp=sharing!!!!!!!!!!!!Links:- makemore on github: github.com/karpathy/makemore- jupyter notebook I built in this video: github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part4_backprop.ipynb- collab notebook: colab.research.google.com/drive/1WV2oi2fh9XXyldh02wupFQX0wh5ZC-z-?usp=sharing- my website: karpathy.ai- my twitter: twitter.com/karpathy- our Discord channel: discord.gg/3zy8kqD9CpSupplementary links:- Yes you should understand backprop: karpathy.medium.com/yes-you-should-understand-backprop-e2f06eab496b- BatchNorm paper: arxiv.org/abs/1502.03167- Bessel’s Correction: http://math.oxford.emory.edu/site/math117/besselCorrection/- Bengio et al. 2003 MLP LM jmlr.org/papers/volume3/bengio03a/bengio03a.pdf Chapters:00:00:00 intro: why you should care & fun history00:07:26 starter code00:13:01 exercise 1: backproping the atomic compute graph01:05:17 brief digression: bessel’s correction in batchnorm01:26:31 exercise 2: cross entropy loss backward pass01:36:37 exercise 3: batch norm layer backward pass01:50:02 exercise 4: putting it all together01:54:24 outro
Building makemore Part 3: Activations & Gradients, BatchNorm Andrej Karpathy 2022-10-04 | We dive into some of the internals of MLPs with multiple layers and scrutinize the statistics of the forward pass activations, backward pass gradients, and some of the pitfalls when they are improperly scaled. We also look at the typical diagnostic tools and visualizations you'd want to use to understand the health of your deep network. We learn why training deep neural nets can be fragile and introduce the first modern innovation that made doing so much easier: Batch Normalization. Residual connections and the Adam optimizer remain notable todos for later video.Links:- makemore on github: github.com/karpathy/makemore- jupyter notebook I built in this video: github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part3_bn.ipynb- collab notebook: colab.research.google.com/drive/1H5CSy-OnisagUgDUXhHwo1ng2pjKHYSN?usp=sharing- my website: karpathy.ai- my twitter: twitter.com/karpathy- Discord channel: discord.gg/3zy8kqD9CpUseful links:- "Kaiming init" paper: arxiv.org/abs/1502.01852- BatchNorm paper: arxiv.org/abs/1502.03167- Bengio et al. 2003 MLP language model paper (pdf): jmlr.org/papers/volume3/bengio03a/bengio03a.pdf- Good paper illustrating some of the problems with batchnorm in practice: arxiv.org/abs/2105.07576Exercises:- E01: I did not get around to seeing what happens when you initialize all weights and biases to zero. Try this and train the neural net. You might think either that 1) the network trains just fine or 2) the network doesn't train at all, but actually it is 3) the network trains but only partially, and achieves a pretty bad final performance. Inspect the gradients and activations to figure out what is happening and why the network is only partially training, and what part is being trained exactly.- E02: BatchNorm, unlike other normalization layers like LayerNorm/GroupNorm etc. has the big advantage that after training, the batchnorm gamma/beta can be "folded into" the weights of the preceeding Linear layers, effectively erasing the need to forward it at test time. Set up a small 3-layer MLP with batchnorms, train the network, then "fold" the batchnorm gamma/beta into the preceeding Linear layer's W,b by creating a new W2, b2 and erasing the batch norm. Verify that this gives the same forward pass during inference. i.e. we see that the batchnorm is there just for stabilizing the training, and can be thrown out after training is done! pretty cool.Chapters:00:00:00 intro00:01:22 starter code00:04:19 fixing the initial loss 00:12:59 fixing the saturated tanh00:27:53 calculating the init scale: “Kaiming init”00:40:40 batch normalization01:03:07 batch normalization: summary01:04:50 real example: resnet50 walkthrough01:14:10 summary of the lecture01:18:35 just kidding: part2: PyTorch-ifying the code01:26:51 viz #1: forward pass activations statistics01:30:54 viz #2: backward pass gradient statistics01:32:07 the fully linear case of no non-linearities01:36:15 viz #3: parameter activation and gradient statistics01:39:55 viz #4: update:data ratio over time01:46:04 bringing back batchnorm, looking at the visualizations01:51:34 summary of the lecture for real this time
Building makemore Part 2: MLP Andrej Karpathy 2022-09-12 | We implement a multilayer perceptron (MLP) character-level language model. In this video we also introduce many basics of machine learning (e.g. model training, learning rate tuning, hyperparameters, evaluation, train/dev/test splits, under/overfitting, etc.).Links:- makemore on github: github.com/karpathy/makemore- jupyter notebook I built in this video: github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part2_mlp.ipynb- collab notebook (new)!!!: colab.research.google.com/drive/1YIfmkftLrz6MPTOO9Vwqrop2Q5llHIGK?usp=sharing- Bengio et al. 2003 MLP language model paper (pdf): jmlr.org/papers/volume3/bengio03a/bengio03a.pdf- my website: karpathy.ai- my twitter: twitter.com/karpathy- (new) Neural Networks: Zero to Hero series Discord channel: discord.gg/3zy8kqD9Cp , for people who'd like to chat more and go beyond youtube commentsUseful links:- PyTorch internals ref http://blog.ezyang.com/2019/05/pytorch-internalsExercises:- E01: Tune the hyperparameters of the training to beat my best validation loss of 2.2- E02: I was not careful with the intialization of the network in this video. (1) What is the loss you'd get if the predicted probabilities at initialization were perfectly uniform? What loss do we achieve? (2) Can you tune the initialization to get a starting loss that is much more similar to (1)?- E03: Read the Bengio et al 2003 paper (link above), implement and try any idea from the paper. Did it work?Chapters:00:00:00 intro00:01:48 Bengio et al. 2003 (MLP language model) paper walkthrough00:09:03 (re-)building our training dataset00:12:19 implementing the embedding lookup table00:18:35 implementing the hidden layer + internals of torch.Tensor: storage, views00:29:15 implementing the output layer00:29:53 implementing the negative log likelihood loss00:32:17 summary of the full network00:32:49 introducing F.cross_entropy and why00:37:56 implementing the training loop, overfitting one batch00:41:25 training on the full dataset, minibatches00:45:40 finding a good initial learning rate00:53:20 splitting up the dataset into train/val/test splits and why01:00:49 experiment: larger hidden layer01:05:27 visualizing the character embeddings01:07:16 experiment: larger embedding size01:11:46 summary of our final code, conclusion01:13:24 sampling from the model01:14:55 google collab (new!!) notebook advertisement
The spelled-out intro to language modeling: building makemore Andrej Karpathy 2022-09-07 | We implement a bigram character-level language model, which we will further complexify in followup videos into a modern Transformer language model, like GPT. In this video, the focus is on (1) introducing torch.Tensor and its subtleties and use in efficiently evaluating neural networks and (2) the overall framework of language modeling that includes model training, sampling, and the evaluation of a loss (e.g. the negative log likelihood for classification).Links:- makemore on github: github.com/karpathy/makemore- jupyter notebook I built in this video: github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part1_bigrams.ipynb- my website: karpathy.ai- my twitter: twitter.com/karpathy- (new) Neural Networks: Zero to Hero series Discord channel: discord.gg/3zy8kqD9Cp , for people who'd like to chat more and go beyond youtube commentsUseful links for practice:- Python + Numpy tutorial from CS231n cs231n.github.io/python-numpy-tutorial . We use torch.tensor instead of numpy.array in this video. Their design (e.g. broadcasting, data types, etc.) is so similar that practicing one is basically practicing the other, just be careful with some of the APIs - how various functions are named, what arguments they take, etc. - these details can vary.- PyTorch tutorial on Tensor pytorch.org/tutorials/beginner/basics/tensorqs_tutorial.html- Another PyTorch intro to Tensor pytorch.org/tutorials/beginner/nlp/pytorch_tutorial.htmlExercises:E01: train a trigram language model, i.e. take two characters as an input to predict the 3rd one. Feel free to use either counting or a neural net. Evaluate the loss; Did it improve over a bigram model?E02: split up the dataset randomly into 80% train set, 10% dev set, 10% test set. Train the bigram and trigram models only on the training set. Evaluate them on dev and test splits. What can you see?E03: use the dev set to tune the strength of smoothing (or regularization) for the trigram model - i.e. try many possibilities and see which one works best based on the dev set loss. What patterns can you see in the train and dev set loss as you tune this strength? Take the best setting of the smoothing and evaluate on the test set once and at the end. How good of a loss do you achieve?E04: we saw that our 1-hot vectors merely select a row of W, so producing these vectors explicitly feels wasteful. Can you delete our use of F.one_hot in favor of simply indexing into rows of W?E05: look up and use F.cross_entropy instead. You should achieve the same result. Can you think of why we'd prefer to use F.cross_entropy instead?E06: meta-exercise! Think of a fun/interesting exercise and complete it.Chapters:00:00:00 intro00:03:03 reading and exploring the dataset00:06:24 exploring the bigrams in the dataset00:09:24 counting bigrams in a python dictionary00:12:45 counting bigrams in a 2D torch tensor ("training the model")00:18:19 visualizing the bigram tensor00:20:54 deleting spurious (S) and (E) tokens in favor of a single . token00:24:02 sampling from the model00:36:17 efficiency! vectorized normalization of the rows, tensor broadcasting 00:50:14 loss function (the negative log likelihood of the data under our model)01:00:50 model smoothing with fake counts01:02:57 PART 2: the neural network approach: intro01:05:26 creating the bigram dataset for the neural net01:10:01 feeding integers into neural nets? one-hot encodings01:13:53 the "neural net": one linear layer of neurons implemented with matrix multiplication01:18:46 transforming neural net outputs into probabilities: the softmax01:26:17 summary, preview to next steps, reference to micrograd01:35:49 vectorized loss01:38:36 backward and update, in PyTorch01:42:55 putting everything together01:47:49 note 1: one-hot encoding really just selects a row of the next Linear layer's weight matrix01:50:18 note 2: model smoothing as regularization loss01:54:31 sampling from the neural net01:56:16 conclusion
Stable diffusion dreams of psychedelic faces Andrej Karpathy 2022-08-19 | Prompt: "psychedelic faces"Stable diffusion takes a noise vector as input and samples an image. To create this video I smoothly (spherically) interpolate between randomly chosen noise vectors and render frames along the way.This video was produced by one A100 GPU taking about 10 tabs and dreaming about the prompt overnight (~8 hours). While I slept and dreamt about other things.Music: Stars by JVNALinks:- Stable diffusion: stability.ai/blog- Code used to make this video: gist.github.com/karpathy/00103b0037c5aaea32fe1da1af553355- My twitter: twitter.com/karpathy
Stable diffusion dreams of steampunk brains Andrej Karpathy 2022-08-17 | Prompt: "ultrarealistic steam punk neural network machine in the shape of a brain, placed on a pedestal, covered with neurons made of gears. dramatic lighting. #unrealengine"Stable diffusion takes a noise vector as input and samples an image. To create this video I smoothly (spherically) interpolate between randomly chosen noise vectors and render frames along the way.This video was produced by one A100 GPU dreaming about the prompt overnight (~8 hours). While I slept and dreamt about other things.This is version 2 video of this prompt, with (I think?) a bit higher quality and trippy AGI music.Music: Wonders by JVNALinks:- Stable diffusion: stability.ai/blog- Code used to make this video: gist.github.com/karpathy/00103b0037c5aaea32fe1da1af553355- My twitter: twitter.com/karpathy
Stable diffusion dreams of tattoos Andrej Karpathy 2022-08-16 | Dreams of tattoos. (There are a few discrete jumps in the video because I had to erase portions that got just a little 🌶️, believe I got most of it)Links- Stable diffusion: stability.ai/blog- Code used to make this video: gist.github.com/karpathy/00103b0037c5aaea32fe1da1af553355- My twitter: twitter.com/karpathy
The spelled-out intro to neural networks and backpropagation: building micrograd Andrej Karpathy 2022-08-16 | This is the most step-by-step spelled-out explanation of backpropagation and training of neural networks. It only assumes basic knowledge of Python and a vague recollection of calculus from high school.Links:- micrograd on github: github.com/karpathy/micrograd- jupyter notebooks I built in this video: github.com/karpathy/nn-zero-to-hero/tree/master/lectures/micrograd- my website: karpathy.ai- my twitter: twitter.com/karpathy- "discussion forum": nvm, use youtube comments below for now :)- (new) Neural Networks: Zero to Hero series Discord channel: discord.gg/3zy8kqD9Cp , for people who'd like to chat more and go beyond youtube commentsExercises:you should now be able to complete the following google collab, good luck!:colab.research.google.com/drive/1FPTx1RXtBfc4MaTkf7viZZD4U2F9gtKN?usp=sharingChapters:00:00:00 intro00:00:25 micrograd overview00:08:08 derivative of a simple function with one input00:14:12 derivative of a function with multiple inputs00:19:09 starting the core Value object of micrograd and its visualization00:32:10 manual backpropagation example #1: simple expression00:51:10 preview of a single optimization step00:52:52 manual backpropagation example #2: a neuron01:09:02 implementing the backward function for each operation01:17:32 implementing the backward function for a whole expression graph01:22:28 fixing a backprop bug when one node is used multiple times01:27:05 breaking up a tanh, exercising with more operations01:39:31 doing the same thing but in PyTorch: comparison01:43:55 building out a neural net library (multi-layer perceptron) in micrograd01:51:04 creating a tiny dataset, writing the loss function01:57:56 collecting all of the parameters of the neural net02:01:12 doing gradient descent optimization manually, training the network02:14:03 summary of what we learned, how to go towards modern neural nets02:16:46 walkthrough of the full code of micrograd on github02:21:10 real stuff: diving into PyTorch, finding their backward pass for tanh02:24:39 conclusion02:25:20 outtakes :)
Stable diffusion dreams of blueberry spaghetti for one night Andrej Karpathy 2022-08-16 | Prompt: "blueberry spaghetti"Stable diffusion takes a noise vector as input and samples an image. To create this video I simply smoothly interpolate between randomly chosen noise vectors and render frames along the way.Links- Stable diffusion: stability.ai/blog- Code used to make this video: gist.github.com/karpathy/00103b0037c5aaea32fe1da1af553355- My twitter: twitter.com/karpathy
Stable diffusion dreams of steam punk neural networks Andrej Karpathy 2022-08-16 | A stable diffusion dream. The prompt was "ultrarealistic steam punk neural network machine in the shape of a brain, placed on a pedestal, covered with neurons made of gears. dramatic lighting. #unrealengine"the new and improved v2 version of this video is now here: youtube.com/watch?v=2oKjtvYslMYgenerated with this hacky script: gist.github.com/karpathy/00103b0037c5aaea32fe1da1af553355The script slowly meanders through noise space to explore the space of possible generations for the fixed prompt.Stable diffusion: stability.ai/blog/stable-diffusion-announcement
