OpenAI Sora and DiTs: Scalable Diffusion Models with Transformers Gabriel Mongaras 2024-02-18 | Sora: openai.com/soraSora paper (Video generation models as world simulators): openai.com/research/video-generation-models-as-world-simulatorsDiTs - Scalable Diffusion Models with Transformers paper: arxiv.org/abs/2212.09748My notes: drive.google.com/file/d/1h2pcgkrI0b6965f1xjf4kTyhhvxZNM3b/view?usp=drive_link
Deterministic Image Editing with DDPM Inversion, DDIM Inversion, Null Inversion and Prompt-to-Prompt Gabriel Mongaras 2024-07-31 | Null-text Inversion for Editing Real Images using Guided Diffusion Models: arxiv.org/abs/2211.09794An Edit Friendly DDPM Noise Space: Inversion and Manipulations: arxiv.org/abs/2304.06140Prompt-to-Prompt Image Editing with Cross Attention Control: arxiv.org/abs/2208.0162600:00 Intro01:24 Current image editing techniques11:42 Deriving DDPM and DDIM23:08 DDIM inversion32:46 Null inversion47:15 DDPM inversion1:01:18 Prompt-to-prompt1:10:52 Conclusion
Attending to Topological Spaces: The Cellular Transformer Gabriel Mongaras 2024-07-22 | Paper here: arxiv.org/abs/2405.14094Notes: drive.google.com/file/d/12g_KkHqXD6mEDILJzYbCC08i8cDHITfC/view?usp=drive_link00:00 Intro01:39 Cellular complexes07:26 K-cochain13:26 Defining structure on the cell20:28 Cellular transformer34:18 Positional encodings and outro
Learning to (Learn at Test Time): RNNs with Expressive Hidden States Gabriel Mongaras 2024-07-12 | Paper here: arxiv.org/abs/2407.04620Code!: github.com/test-time-training/ttt-lm-pytorchNotes: drive.google.com/file/d/127a1UBm_IN_WMKG-DmEvfJ8Pja-9BwDk/view?usp=drive_link00:00 Intro04:40 Problem with RNNs06:38 Meta learning and method idea09:13 Update rule and RNN inner loop15:07 Learning the loss function outer loop21:21 Parallelizing training30:05 Results
WARP: On the Benefits of Weight Averaged Rewarded Policies Gabriel Mongaras 2024-07-06 | Paper here: arxiv.org/abs/2406.16768Notes: drive.google.com/file/d/11UK7mEZwNVUMYuXwvOTfaqHhN8zSYm5M/view?usp=drive_link00:00 Intro and RLHF17:30 Problems with RLHF21:08 Overview of their method23:47 EMA28:00 Combining policies with SLERP37:34 Linear interpolation towards initialization40:32 Code44:16 Results
CoDeF: Content Deformation Fields for Temporally Consistent Video Processing Gabriel Mongaras 2024-06-25 | Paper: arxiv.org/abs/2308.07926Paper page: qiuyu96.github.io/CoDeFCode: github.com/qiuyu96/CoDeFMy notes: drive.google.com/file/d/10PMKdd5XBd6Y60HlRB9IW9naR2bWziDT/view?usp=drive_link00:00 Intro03:00 Method overview08:40 Method details15:24 Tricks done for training and how to actually train this thing19:24 Flow loss and masking25:10 Conclusion
Mamba 2 - Transformers are SSMs: Generalized Models and Efficient Algorithms Through SSS Duality Gabriel Mongaras 2024-06-16 | Paper here: arxiv.org/abs/2405.21060Code!: github.com/state-spaces/mamba/blob/main/mamba_ssm/modules/mamba2.pyNotes: drive.google.com/file/d/1--XGPFeXQyx4CPxgYjzR4qrLd-baLWQC/view?usp=sharing00:00 Intro01:45 SSMs08:00 Quadratic form of an SSM15:02 Expanded form of an SSM24:00 Attention - it's all you need??29:55 Kernel attention32:50 Linear attention34:32 Relating attention to SSMs38:35 Defining the M matrix43:48 Splitting the M matrix46:30 Off diagonal decomposition54:00 Recurrent form of the off diagonal1:03:30 Combining the M matrix blocks and code1:06:22 Complexity and other analysis
CoPE - Contextual Position Encoding: Learning to Count Whats Important Gabriel Mongaras 2024-06-04 | Paper: arxiv.org/abs/2405.18719My notes: drive.google.com/file/d/1y9VHZc7MLqc6t2SHHdlVTYeW3czmmRbl/view?usp=sharing00:00 Intro02:44 Background09:58 CoPE24:50 Code32:16 Results
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models Gabriel Mongaras 2024-05-28 | Paper: arxiv.org/abs/2403.03100Demo: speechresearch.github.io/naturalspeech3Code: huggingface.co/spaces/amphion/naturalspeech3_facodecMy notes: drive.google.com/file/d/1xnzErd_86B6eLwqpLckhoEQKqkxFPyM_/view?usp=drive_link00:00 Intro05:34 Architecture overview18:45 GRL and subspace independence24:45 Discrete diffusion Model41:00 factorized diffusion model44:00 Conclusion and results
xLSTM: Extended Long Short-Term Memory Gabriel Mongaras 2024-05-17 | Paper: arxiv.org/abs/2405.04517My notes: drive.google.com/file/d/1wFYvU_1oUWcCNuQ91zTpSGAeNUsPjlt3/view?usp=drive_link00:00 Intro05:44 LSTM13:38 Problems paper addresses14:12 sLSTM23:00 sLSTM Memory mixing27:08 mLSTM35:14 Results and stuff
KAN: Kolmogorov-Arnold Networks Gabriel Mongaras 2024-05-04 | Paper: arxiv.org/abs/2404.19756Spline Video: https://m.youtube.com/watch?v=qhQrRCJ-mVgMy notes: drive.google.com/file/d/1twcIF13nG8Qc10_qeDqCZ4NaUh9tFsAH/view?usp=drive_link00:00 Intro00:45 MLPs and Intuition05:12 Splines19:02 KAN Formulation28:00 Potential Downsides to KANs32:09 Results
LADD: Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation Gabriel Mongaras 2024-04-29 | Paper: arxiv.org/abs/2403.12015My notes: drive.google.com/file/d/1s1-nnWR_ZR26PNSAoZR1Xj3nuD9UZlvR/view?usp=sharing00:00 Intro01:31 Diffusion Models08:08 Latent Diffusion Models10:04 Distillation12:02 Aversarial Diffusion Distillation (ADD)17:06 Latent Aversarial Diffusion Distillation (LADD)22:20 Results
Visual AutoRegressive Modeling:Scalable Image Generation via Next-Scale Prediction Gabriel Mongaras 2024-04-21 | Paper: arxiv.org/abs/2404.02905Demo: https://var.vision/Code: github.com/FoundationVision/VARMy notes: drive.google.com/file/d/1qym3JG-0xqEgQhdvkt9N17o-ZzUWy2sn/view?usp=drive_link00:00 Intro00:53 DiTs04:06 Autoregressive Image Transformers06:23 Tokenization problem with AR ViTs08:43 VAE10:47 Discrete Quantization - VQGAN16:42 Visual Autoregressive Modeling21:31 Causal Inference with VAR24:02 Losses25:16 Residual Modeling33:26 Summary34:11 Results
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention Gabriel Mongaras 2024-04-14 | Paper: arxiv.org/abs/2404.07143My notes: drive.google.com/file/d/1plWJDwHTZkRK9PDdvaLMnZjFR6fVvNLH/view?usp=drive_link00:00 Intro07:17 Model intuition11:00 Memory retrieval operation16:29 Hidden state updates21:58 Delta update24:10 Is it causal?25:26 Combining local attention and RNN27:26 Results30:25 Sampling and Conclusion
Mixture-of-Depths: Dynamically allocating compute in transformer-based language models Gabriel Mongaras 2024-04-08 | Paper: arxiv.org/abs/2404.02258My notes: drive.google.com/file/d/1o4v5te1yfuK_FQPvvS8SR55Sysg04dYK/view?usp=drive_link00:00 Intro06:02 Mixture of Experts (MoE)15:12 Mixture of Depths (MoD)17:04 The gradients must flow!22:40 Autoregressive Sampling33:58 Results
Q* AGI Achieved (Apr Fools) Gabriel Mongaras 2024-04-01 | Q* paper link: link.springer.com/content/pdf/10.1007/BF00992698.pdfApril fools 😏
Stable Diffusion 3: Scaling Rectified Flow Transformers for High-Resolution Image Synthesis Gabriel Mongaras 2024-03-28 | Website paper: stability.ai/news/stable-diffusion-3-research-paperPaper: arxiv.org/abs/2403.03206My notes: drive.google.com/file/d/1n8rSM3OuOkzDBlXdK5VBrnADnEXp4xXv/view?usp=drive_link00:00 Intro01:58 DDPM13:16 ODE/SDE formulation and score18:09 ODE intuition21:38 Rectified Flows27:46 Sampling from a diffusion model29:16 Going to the latent space32:17 CLIP37:53 Model architecture56:18 Results and stuff
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection Gabriel Mongaras 2024-03-21 | My notes: drive.google.com/file/d/1l2B4m8tDVchfsplIbps4-9533fcxqubF/view?usp=drive_linkPaper: arxiv.org/abs/2403.0350700:00 Intro02:44 Intuition and proof of low rank12:28 GaLore intuition16:38 More GaLore intuition21:20 GaLore algorithm27:50 Algorithm analysis33:00 Results
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits and BitNet Gabriel Mongaras 2024-03-06 | My notes:BitNet: drive.google.com/file/d/1iA2tISamkfQq4jgZZBBSH1MN3Bgtc99_/view?usp=sharingEra of 1-bit LLMs: drive.google.com/file/d/1iNy91MTP53kTCSkeqHBqMOSePPyoYvCD/view?usp=sharingBitNet: Scaling 1-bit Transformers for Large Language Models: arxiv.org/abs/2310.11453The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits: arxiv.org/abs/2402.1776400:00 Intro03:10 BitLinear Intuition08:05 Weight Quantization10:35 Activation Quantization16:30 Matrix Multiplication and Dequantizing23:08 Model Parallelism with Group Quantization and Normalization32:36 Other Training Stuff37:11 BitNet Results39:11 The Era of 1-Bit LLMs
DoRA: Weight-Decomposed Low-Rank Adaptation Gabriel Mongaras 2024-02-23 | Paper: arxiv.org/abs/2402.09353My notes: drive.google.com/file/d/1hA56lNtz7jxQPWIxBpnDUsiLFaFZlyyP/view?usp=sharing
A Decoder-only Foundation Model For Time-series Forecasting Gabriel Mongaras 2024-02-07 | Paper: arxiv.org/abs/2310.10688Notes: drive.google.com/file/d/1fmk5Z5VJkqHvEbNXlq1OiIBP317NqNfN/view?usp=sharing
Lumiere: A Space-Time Diffusion Model for Video Generation Gabriel Mongaras 2024-02-02 | Paper: arxiv.org/abs/2401.12945Demo: lumiere-video.github.ioNotes: drive.google.com/file/d/1fJl-ijVy6KML1YwM_9UVVU-MSfipDIqe/view?usp=sharing
Exphormer: Sparse Transformers for Graphs Gabriel Mongaras 2024-01-29 | Paper here: arxiv.org/abs/2303.06147Notes: drive.google.com/file/d/1eXoXtPgJYKBTKd7oN8StuBLW453yWJ3f/view?usp=drive_link
Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads Gabriel Mongaras 2024-01-24 | Paper here: arxiv.org/abs/2401.10774demo: sites.google.com/view/medusa-llmNotes: drive.google.com/file/d/1eOminZIC4wrjjWIBnSroxBYduCXzs86E/view?usp=drive_link
Boundary Attention: Learning to Find Faint Boundaries at Any Resolution Gabriel Mongaras 2024-01-18 | Paper here: arxiv.org/abs/2401.00935Notes: drive.google.com/file/d/1eAiAhbmvczYQwHqHHv-GJeDlX-WlZBBI/view?usp=sharing
Cached Transformers: Improving Transformers with Differentiable Memory Cache Gabriel Mongaras 2024-01-04 | Paper here: arxiv.org/abs/2312.12742Code here: github.com/annosubmission/GRC-CacheNotes: drive.google.com/file/d/1cgR14tZmrF3lQROMT_2RUig2dBfhqU9z/view?usp=sharing
Translatotron 3: Speech to Speech Translation with Monolingual Data Gabriel Mongaras 2023-12-27 | Translatotron 3: arxiv.org/abs/2305.17547Translatotron 2: arxiv.org/abs/2107.08661Demo: google-research.github.io/lingvo-lab/translatotron3Notes: Translatotron 3: drive.google.com/file/d/1EfOCuKp9yeLBzhxjsiTWuoYaVToBbgon/view?usp=sharingTranslatotron 2: drive.google.com/file/d/1zPrIvZspMWpWPaFhvgpM2DEYzsvTL8R6/view?usp=drive_link
Mamba: Linear-Time Sequence Modeling with Selective State Spaces Gabriel Mongaras 2023-12-12 | Paper here: arxiv.org/abs/2312.00752The annotated S4: srush.github.io/annotated-s4Notes: drive.google.com/file/d/1aoaKj3kuTtpHi0OzinXZGyZIFxhqp514/view?usp=sharing
Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference Gabriel Mongaras 2023-12-06 | Paper Link: arxiv.org/abs/2310.04378My Notes: drive.google.com/file/d/1aUDxMSWNAqMkg0P91Ms1vu4yCeTtzSsR/view?usp=sharing
Adversarial Diffusion Distillation Gabriel Mongaras 2023-11-30 | Paper Link: arxiv.org/abs/2311.17042Stability Link: stability.ai/research/adversarial-diffusion-distillationMy Notes: drive.google.com/file/d/1a7EZpQ-4_jjt7Fic1EQlyGnHOX1xB9Af/view?usp=sharing
Unsupervised Discovery of Semantic Latent Directions in Diffusion Models Gabriel Mongaras 2023-11-21 | Paper found here: arxiv.org/abs/2302.12469My notes: drive.google.com/file/d/1_wFtrtxZk7ZYq6-FfUILET3Nga8KCzsz/view?usp=drive_link
DALL-E 3 - Improving Image Generation with Better Captions Gabriel Mongaras 2023-11-20 | Blog post here: openai.com/dall-e-3My notes: drive.google.com/file/d/1_lSM24dNSdzAvP8MKaKIyfbsASn4UfYe/view?usp=sharing
LRM: Large Reconstruction Model for Single Image to 3D Gabriel Mongaras 2023-11-13 | Paper found here: arxiv.org/abs/2311.04400My notes: drive.google.com/file/d/1_cI6cYIm8QZrv0lhfYBG7ULXc4szr8Hg/view?usp=sharing
CodeFusion: A Pre-trained Diffusion Model for Code Generation Gabriel Mongaras 2023-11-06 | Paper found here: arxiv.org/abs/2310.17680v1My chicken scratch: drive.google.com/file/d/1ErA6RsKW__uxmlprgdIO13PRCU69Q-U5/view?usp=drive_link
Matryoshka Diffusion Models Explained Gabriel Mongaras 2023-10-30 | Paper found here: arxiv.org/abs/2310.15111
UniAudio: An Audio Foundation Model Toward Universal Audio Generation Gabriel Mongaras 2023-10-22 | Paper: arxiv.org/abs/2310.00704Code: github.com/yangdongchao/UniAudioDemo: https://dongchaoyang.top/UniAudio_demo/
QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models Gabriel Mongaras 2023-10-16 | Paper found here: arxiv.org/abs/2309.14717v2
StreamingLLM - Efficient Streaming Language Models with Attention Sinks Explained Gabriel Mongaras 2023-10-07 | Paper found here: arxiv.org/abs/2309.17453Code found here: github.com/mit-han-lab/streaming-llm
FreeU: Free Lunch in Diffusion U-Net Explained Gabriel Mongaras 2023-09-24 | Paper found here: arxiv.org/abs/2309.11497
InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation Explained Gabriel Mongaras 2023-09-17 | Paper found here: arxiv.org/abs/2309.06380
Llama/Wizard LM Finetuning with Huggingface on RunPod Gabriel Mongaras 2023-09-16 | A demo I made to show how to fine-tune a WizardLM model with Huggingface and peft.Presentation: docs.google.com/presentation/d/17TyDtImkcXnIXwd6CDoYxCXBprvtD_n1I3RlImJg8gQ/edit?usp=sharingGithub: github.com/gmongaras/Wizard_QLoRA_Finetuning
2x Faster Language Model Pre-training via Masked Structural Growth Gabriel Mongaras 2023-09-10 | Paper found here: arxiv.org/abs/2305.02869
Bayesian Flow Networks (BFN) Explained Gabriel Mongaras 2023-09-03 | Paper found here: arxiv.org/abs/2308.07037
WizardLM: Empowering Large Language Models to Follow Complex Instructions Explained Gabriel Mongaras 2023-08-27 | Paper found here: arxiv.org/abs/2304.12244Code release: github.com/nlpxucan/WizardLM
From Sparse to Soft Mixtures of Experts Explained Gabriel Mongaras 2023-08-21 | Paper found here: arxiv.org/abs/2308.00951
BK-SDM: Architecturally Compressed Stable Diffusion for Efficient T2I Generation Explained Gabriel Mongaras 2023-08-16 | Paper found here: openreview.net/forum?id=bOVydU0XKC
Direct Preference Optimization (DPO): Your Language Model is Secretly a Reward Model Explained Gabriel Mongaras 2023-08-10 | Paper found here: arxiv.org/abs/2305.18290
Universal and Transferable Adversarial Attacks on Aligned Language Models Explained Gabriel Mongaras 2023-08-06 | Paper found here: arxiv.org/abs/2307.15043Demo here: llm-attacks.org
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis Explained Gabriel Mongaras 2023-08-01 | Paper found here: arxiv.org/abs/2307.01952
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations Explained Gabriel Mongaras 2023-07-30 | Paper found here: arxiv.org/abs/2108.01073
