Uploaded April 2025 | Updated September 2026, 1 week ago

---

Randall Balestriero (Meta AI) shares three recent results that each push back on conventional wisdom in ML.

First, the headline finding: if you take a 7-billion-parameter language model, initialize it randomly, and train it from scratch on just 20,000 labeled examples for a classification task like sentiment analysis, it works. Stable training curves, minimal overfitting, performance that matches LoRA-finetuned pre-trained models. The obvious question -- is months of expensive pre-training on internet-scale data actually worth it? -- gets a surprisingly qualified answer. For narrow discriminative tasks, random initialization is competitive. Pre-training still wins for generation and open-ended reasoning, but there is a whole spectrum between the two extremes that nobody is really exploring yet.

Second, a theoretical result with Yann LeCun proving that self-supervised and supervised learning objectives are mathematically equivalent up to how you define the label structure. SSL does not learn better representations because of its loss function; it learns them because it uses finer-grained pairwise relationships instead of collapsing all cars into "car." This equivalence lets you port decades of supervised learning theory -- class imbalance corrections, neural collapse results, semi-supervised weighting -- directly into SSL, and Randall walks through how VICReg falls out naturally from a least-squares supervised objective under this framework.

Third, a fairness audit of implicit neural representations used for earth/climate data. Models that look accurate on average turn out to be nearly random around islands and coastlines -- exactly the places where policy decisions about climate adaptation matter most. The culprit is partly architectural: Fourier bases assume stationarity, and switching to wavelets recovers some of the lost localization. But the deeper problem is data bias, including the same geographic skew Mark Ibrahim documented in ImageNet, where most training images come from North America.

SPONSOR MESSAGES:
***
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto tufalabs.ai
***

---
TIMESTAMPS:
00:00:00 Random Initialization Rivals Pre-Training
00:01:29 Is Next-Token Prediction Worth the Cost?
00:04:44 What Do These Models Actually Learn?
00:07:59 Build Specialist LLMs Like It Is 2019
00:10:31 The Fair Language Model Paradox
00:13:38 Benchmarks, Generation, and Understanding
00:16:04 The Birth of Self-Supervised Learning
00:19:14 Class Balance, VICReg, and Unifying Representation Learning
00:25:18 No Location Left Behind: Fairness in Earth Models
00:30:24 Policy, Accountability, and Crowdsourced Data Bias

---
REFERENCES:
[00:00:00] Is LLM Pre-Training by Next Token Prediction Worth the Cost? sslneurips24.github.io
[00:05:46] Lottery Ticket Hypothesis arxiv.org/abs/1803.03635
[00:10:31] The Fair Language Model Paradox arxiv.org/abs/2410.11985
[00:16:04] The Birth of Self-Supervised Learning openreview.net/forum?id=NhYAjAAdQT
[00:19:14] VICReg arxiv.org/abs/2105.04906
[00:25:18] No Location Left Behind arxiv.org/abs/2502.06831
[00:33:14] Geographic bias in large visual models arxiv.org/abs/2304.12210

---
LINKS:
Full Transcript: app.rescript.info/share/1fecffe43479a465c6b19622356faf8f
Download PDF transcript: app.rescript.info/api/public/sessions/8f1ca777a45ad475/pdf