Lecture 13: Ring Attention @GPUMODE

Lecture 13: Ring Attention @GPUMODE

Uploaded April 2024 | Updated September 2026, 3 hours ago

Slides docs.google.com/presentation/d/180lS8XbeR1_bTMaldg21LKYQkjXftHuh9VnZ3xk27qQ/edit
Code github.com/cuda-mode/ring-attention

Lecture 13: Ring Attention

FastVideo: Accelerating large video diffusion models

Lecture 61: D-Matrix Corsair

tvm-ffi

Lecture 65: Neighborhood Attention

Smol Training Playbook

Mojo: Modular’s unified device accelerator language

Livestream KernelBot: Benchmark GPU Kernels on Discord

Lecture 12: Flash Attention

Getting Started with CuTe DSL

Lecture 67: NCCL and NVSHMEM

Lecture 13: Ring Attention