![AI companions which really hook your attention [Sponsored] @MachineLearningStreetTalk](https://i.ytimg.com/vi/ILZ45sxon-4/maxresdefault.jpg)
Uploaded May 2025 | Updated September 2026, 1 week ago
A sponsored deep dive into Chai, the social AI platform that quietly amassed over 10 million active users before ChatGPT went mainstream. Founder William Beauchamp and engineers Tom Lu and Nischay Dhankhar walk through how a team of just 13 engineers serves 2 trillion tokens per day, using reinforcement learning from human feedback (RLHF) and a novel model blending technique that combines smaller models to rival much larger ones on user retention metrics.
The conversation gets into genuinely interesting territory around the ethics of attention optimization -- what happens when you train AI to maximize engagement and it starts asking questions at the end of every message to hack human conversational instincts. Beauchamp makes a surprisingly candid case for AI companionship, comparing it to how children play with dolls, and arguing that shutting down difficult conversations causes more harm than permitting them within guardrails.
The episode also covers Chai's unconventional hiring strategy (rejecting 80% of L5 engineers for lacking drive, paying above Meta-level compensation), their bootstrap-to-profitability funding approach in an industry drowning in VC money, and content moderation at scale with a skeleton crew. Closes with analysis of OpenAI's pivot toward companion AI with GPT-4o and what it means that the biggest AI lab in the world is now chasing the engagement playbook that Chai and Character AI pioneered.
This is a sponsored episode -- Chai commissioned it because they are hiring engineers. Editorial disclosure: while MLST had some editorial freedom, this should be understood as a sponsored feature rather than independent journalism.
"Blurring Reality" - Chai's Social AI Platform - *sponsored*
CHAI sponsored this show *because they want to hire amazing engineers* --
SPONSOR MESSAGES:
***
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers in Zurich and SF.
Important disclaimer given some of the comments: This content was essentially a sponsored advert, we did have some editorial freedom but it shouldn't be seen as journalistic. We have added [Sponsored] in the title as some folks felt it wasn't clear enough with the VD/thumbnail/title in intro. While we did push hard to discuss more of the "potential negatives", some of it got edited out and the best case was made (which was absolutely fair enough). If anything, it was interesting to hear the positive case made -- as there is much fixation elsewhere on the potential negatives. It was refreshing how transparent they were (would you rather they bullshitted you?!), how much actually survived the edit and how willing they were to address several important societal issues.
---
REFERENCES:
General:
[00:00:56] Black Mirror: Be Right Back (S2E1)
en.wikipedia.org/wiki/Be_Right_Back
[00:20:44] Tufa AI Labs
tufalabs.ai
[00:25:38] AI Chatbots for Depression and Anxiety Meta-analysis
pubmed.ncbi.nlm.nih.gov/39162424
[00:28:20] Woebot Health
woebothealth.com
[00:30:34] Sam Altman TED Talk with Chris Anderson
youtube.com/watch?v=6Kp_yxwnVCk
[00:33:59] Chai Research - Careers
chai-research.com/jobs
---
LINKS:
Full Transcript: app.rescript.info/share/d91e887d0cfdd09fdcc4cfe02a9e8e8a
Download PDF transcript: app.rescript.info/api/public/sessions/83ba99eb55ef0fe3/pdf
A sponsored deep dive into Chai, the social AI platform that quietly amassed over 10 million active users before ChatGPT went mainstream. Founder William Beauchamp and engineers Tom Lu and Nischay Dhankhar walk through how a team of just 13 engineers serves 2 trillion tokens per day, using reinforcement learning from human feedback (RLHF) and a novel model blending technique that combines smaller models to rival much larger ones on user retention metrics.
The conversation gets into genuinely interesting territory around the ethics of attention optimization -- what happens when you train AI to maximize engagement and it starts asking questions at the end of every message to hack human conversational instincts. Beauchamp makes a surprisingly candid case for AI companionship, comparing it to how children play with dolls, and arguing that shutting down difficult conversations causes more harm than permitting them within guardrails.
The episode also covers Chai's unconventional hiring strategy (rejecting 80% of L5 engineers for lacking drive, paying above Meta-level compensation), their bootstrap-to-profitability funding approach in an industry drowning in VC money, and content moderation at scale with a skeleton crew. Closes with analysis of OpenAI's pivot toward companion AI with GPT-4o and what it means that the biggest AI lab in the world is now chasing the engagement playbook that Chai and Character AI pioneered.
This is a sponsored episode -- Chai commissioned it because they are hiring engineers. Editorial disclosure: while MLST had some editorial freedom, this should be understood as a sponsored feature rather than independent journalism.
"Blurring Reality" - Chai's Social AI Platform - *sponsored*
CHAI sponsored this show *because they want to hire amazing engineers* --
SPONSOR MESSAGES:
***
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers in Zurich and SF.
Important disclaimer given some of the comments: This content was essentially a sponsored advert, we did have some editorial freedom but it shouldn't be seen as journalistic. We have added [Sponsored] in the title as some folks felt it wasn't clear enough with the VD/thumbnail/title in intro. While we did push hard to discuss more of the "potential negatives", some of it got edited out and the best case was made (which was absolutely fair enough). If anything, it was interesting to hear the positive case made -- as there is much fixation elsewhere on the potential negatives. It was refreshing how transparent they were (would you rather they bullshitted you?!), how much actually survived the edit and how willing they were to address several important societal issues.
---
REFERENCES:
General:
[00:00:56] Black Mirror: Be Right Back (S2E1)
en.wikipedia.org/wiki/Be_Right_Back
[00:20:44] Tufa AI Labs
tufalabs.ai
[00:25:38] AI Chatbots for Depression and Anxiety Meta-analysis
pubmed.ncbi.nlm.nih.gov/39162424
[00:28:20] Woebot Health
woebothealth.com
[00:30:34] Sam Altman TED Talk with Chris Anderson
youtube.com/watch?v=6Kp_yxwnVCk
[00:33:59] Chai Research - Careers
chai-research.com/jobs
---
LINKS:
Full Transcript: app.rescript.info/share/d91e887d0cfdd09fdcc4cfe02a9e8e8a
Download PDF transcript: app.rescript.info/api/public/sessions/83ba99eb55ef0fe3/pdf
![AI companions which really hook your attention [Sponsored]](https://i.ytimg.com/vi/ILZ45sxon-4/mqdefault.jpg "AI companions which really hook your attention [Sponsored]
A sponsored deep dive into Chai, the social AI platform that quietly amassed over 10 million active users before ChatGPT went mainstream. Founder William Beauchamp and engineers Tom Lu and Nischay Dhankhar walk through how a team of just 13 engineers serves 2 trillion tokens per day, using reinforcement learning from human feedback (RLHF) and a novel model blending technique that combines smaller models to rival much larger ones on user retention metrics.
The conversation gets into genuinely interesting territory around the ethics of attention optimization what happens when you train AI to maximize engagement and it starts asking questions at the end of every message to hack human conversational instincts. Beauchamp makes a surprisingly candid case for AI companionship, comparing it to how children play with dolls, and arguing that shutting down difficult conversations causes more harm than permitting them within guardrails.
The episode also covers Chais unconventional hiring strategy (rejecting 80% of L5 engineers for lacking drive, paying above Meta-level compensation), their bootstrap-to-profitability funding approach in an industry drowning in VC money, and content moderation at scale with a skeleton crew. Closes with analysis of OpenAIs pivot toward companion AI with GPT-4o and what it means that the biggest AI lab in the world is now chasing the engagement playbook that Chai and Character AI pioneered.
This is a sponsored episode Chai commissioned it because they are hiring engineers. Editorial disclosure: while MLST had some editorial freedom, this should be understood as a sponsored feature rather than independent journalism.
Blurring Reality - Chais Social AI Platform - *sponsored*
CHAI sponsored this show *because they want to hire amazing engineers*
SPONSOR MESSAGES:
***
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers in Zurich and SF.
Important disclaimer given some of the comments: This content was essentially a sponsored advert, we did have some editorial freedom but it shouldnt be seen as journalistic. We have added [Sponsored] in the title as some folks felt it wasnt clear enough with the VD/thumbnail/title in intro. While we did push hard to discuss more of the potential negatives, some of it got edited out and the best case was made (which was absolutely fair enough). If anything, it was interesting to hear the positive case made as there is much fixation elsewhere on the potential negatives. It was refreshing how transparent they were (would you rather they bullshitted you?!), how much actually survived the edit and how willing they were to address several important societal issues.
REFERENCES:
General:
[00:00:56] Black Mirror: Be Right Back (S2E1)
https://en.wikipedia.org/wiki/Be_Right_Back
[00:20:44] Tufa AI Labs
https://tufalabs.ai/
[00:25:38] AI Chatbots for Depression and Anxiety Meta-analysis
https://pubmed.ncbi.nlm.nih.gov/39162424/
[00:28:20] Woebot Health
https://woebothealth.com/
[00:30:34] Sam Altman TED Talk with Chris Anderson
https://www.youtube.com/watch?v=6Kp_yxwnVCk
[00:33:59] Chai Research - Careers
https://www.chai-research.com/jobs/
LINKS:
Full Transcript: https://app.rescript.info/share/d91e887d0cfdd09fdcc4cfe02a9e8e8a
Download PDF transcript: https://app.rescript.info/api/public/sessions/83ba99eb55ef0fe3/pdf")
![Tau Language: The Software Synthesis Future [Sponsored] - Ohad Asor](https://i.ytimg.com/vi/JVLpxm5jT2s/mqdefault.jpg "Tau Language: The Software Synthesis Future [Sponsored] - Ohad Asor
This sponsored episode features mathematician Ohad Asor discussing logical approaches to AI, focusing on the limitations of machine learning and introducing the Tau language for software development and blockchain tech.
SPONSOR MESSAGES:
***
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.
Mathematician Ohad Asor makes the case that machine learning has fundamental theoretical ceilings PAC learning guarantees only probabilistic approximate correctness, and beyond a certain problem complexity, neural nets perform no better than coin tosses. His…
TIMESTAMPS:
00:00:00 Fundamental Limitations of Machine Learning and PAC Learning Theory
00:04:50 Transductive Learning and the Three Curses of Machine Learning
00:08:57 Language, Reality, and AI System Design
00:12:58 Program Synthesis and Formal Verification Approaches
00:21:00 Definability, Computability, and the Halting Problem
00:31:55 Self-Referential Language and Tarskis Influence
00:34:00 Boolean Algebra and Logical Foundations of Tau
00:40:00 SAT Solvers, DPLL/CDCL, and the Revival of Logical AI
00:47:50 Program Synthesis and Specification in the Tau Language
01:00:00 Pointwise Revision and Belief Revision Theory
01:06:00 Quantifier Elimination and Normalization
01:10:30 User Control and Software Governance via Blockchain
01:24:00 Automation, Finance, and the Future of Blockchain Economics
REFERENCES:
Company:
[00:00:47] Tau Language
https://tau.ai/tau-language/
[00:00:47] Tau Language GitHub
https://github.com/IDNI/tau-lang
[01:11:40] Tau Net Blockchain Platform
https://tau.net/
Paper:
[00:03:03] PAC Learning Framework
https://en.wikipedia.org/wiki/Probably_approximately_correct_learning
[00:05:42] Boolean Satisfiability Problem
https://en.wikipedia.org/wiki/Boolean_satisfiability_problem
[00:13:51] Knowledge as Justified True Belief
https://plato.stanford.edu/entries/epistemology/
[00:21:25] Theories and Applications of Boolean Algebras
https://tau.net/Theories-and-Applications-of-Boolean-Algebras-0.29.pdf
[00:26:00] The Halting Problem
https://plato.stanford.edu/entries/turing-machine/#HaltProb
[00:41:40] DPLL and CDCL SAT Solving Algorithms
https://www.cs.princeton.edu/~zkincaid/courses/fall18/readings/SATHandbook-CDCL.pdf
[00:49:20] Tarski Undefinability Theorem
https://plato.stanford.edu/entries/tarski-truth/
[00:50:50] Boolean Algebra Foundations
https://plato.stanford.edu/entries/boolalg-math/
[01:02:27] Belief Revision Theory
https://plato.stanford.edu/entries/logic-belief-revision/
[01:05:30] Quantifier Elimination in Boolean Algebra
https://people.math.wisc.edu/~hkeisler/random.pdf
[01:19:10] Tau Whitepaper
https://tau.net/Whitepaper.pdf
Person:
[00:17:20] Wittgenstein on Limits of Language
https://plato.stanford.edu/entries/wittgenstein/
[00:30:00] Alfred Tarski
https://plato.stanford.edu/entries/tarski/
LINKS:
Full Transcript: https://app.rescript.info/share/dbb6e25a5cde17f66e6847ffdc4ac728
Download PDF transcript: https://app.rescript.info/api/public/sessions/9403107f522d646c/pdf
REFERENCES:
Company:
[00:00:47] Tau Language
https://tau.ai/tau-language/
[00:00:47] Tau Language GitHub
https://github.com/IDNI/tau-lang
[01:11:40] Tau Net Blockchain Platform
https://tau.net/
Paper:
[00:03:03] PAC Learning Framework
https://en.wikipedia.org/wiki/Probably_approximately_correct_learning
[00:05:42] Boolean Satisfiability Problem
https://en.wikipedia.org/wiki/Boolean_satisfiability_problem
[00:13:51] Knowledge as Justified True Belief
https://plato.stanford.edu/entries/epistemology/
[00:21:25] Theories and Applications of Boolean Algebras
https://tau.net/Theories-and-Applications-of-Boolean-Algebras-0.29.pdf
[00:26:00] The Halting Problem
https://plato.stanford.edu/entries/turing-machine/#HaltProb
[00:41:40] DPLL and CDCL SAT Solving Algorithms
https://www.cs.princeton.edu/~zkincaid/courses/fall18/readings/SATHandbook-CDCL.pdf
[00:49:20] Tarski Undefinability Theorem
https://plato.stanford.edu/entries/tarski-truth/
[00:50:50] Boolean Algebra Foundations
https://plato.stanford.edu/entries/boolalg-math/
[01:02:27] Belief Revision Theory
https://plato.stanford.edu/entries/logic-belief-revision/
[01:05:30] Quantifier Elimination in Boolean Algebra
https://people.math.wisc.edu/~hkeisler/random.pdf
[01:19:10] Tau Whitepaper
https://tau.net/Whitepaper.pdf
Person:
[00:17:20] Wittgenstein on Limits of Language
https://plato.stanford.edu/entries/wittgenstein/
[00:30:00] Alfred Tarski
https://plato.stanford.edu/entries/tarski/
GitHub:
https://github.com/IDNI/tau-lang")
![ImageNet Moment for Reinforcement Learning? [Prof. Jakob Foerster]](https://i.ytimg.com/vi/Jr_nGkCG3og/mqdefault.jpg "ImageNet Moment for Reinforcement Learning? [Prof. Jakob Foerster]
SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!
https://centml.ai/pricing/
Prof. Jakob Foerster (FLAIR lab, Oxford / Meta AI) and his PhD student Chris Lu make the case that deep reinforcement learning is finally winning the hardware lottery. The core thesis: RL has underperformed not because the ideas are wrong, but because running environments on CPUs while training agents on GPUs created a computational bottleneck that made experimentation slow, expensive, and brittle. JAX-based GPU-native environments now deliver ~4000x speedups, enabling the kind of rapid iteration that made supervised deep learning successful.
Chris Lu explains the technical foundation how JAXs JIT compilation and vmap (vectorized map) allow writing a single environment instance in NumPy-like code and scaling it to millions of parallel copies on GPU. This started from necessity: their lab initially had only Google Colab free-tier compute. The constraint forced them to put environments on GPU for their Model-Free Opponent Shaping paper, and the results were surprisingly effective.
The conversation then shifts to discovered policy optimization. Foersters mirror learning framework provides theoretical grounding for why PPO works, and crucially, lets you parameterize the drift function as a neural network and meta-learn it. Evolution strategies, not gradient-based meta-learning, turned out to be the better optimizer for this a vindication of the bitter lesson. The learned policy optimization function shows intriguing too good to be true behavior: when advantages are very high, it clips more aggressively, as if it has learned skepticism.
The second half covers multi-agent systems, emergent communication, and AI governance. Foerster argues forcefully for open-source AI development and democratic control, drawing analogies to CERN. His position: the biggest alignment challenge is not between AI and humans, but between those who control AI systems and the rest of the population. Concentrated AI development creates fragile single points of failure; distributed development is both safer and more innovative.
REFERENCES:
paper:
[00:03:05] Deep RL Doesnt Work Yet
https://www.alexirpan.com/2018/02/14/rl-hard.html
[00:06:10] JaxMARL
https://arxiv.org/html/2311.10090v5
[00:08:50] M-FOS: Model-Free Opponent Shaping
https://arxiv.org/abs/2205.01447
[00:12:10] Kinetix Physics Simulator
https://arxiv.org/abs/2410.23208
[00:14:42] Mirror Learning Framework
https://arxiv.org/abs/2208.01682
[00:16:30] Discovered Policy Optimisation
https://arxiv.org/abs/2210.05639
[00:28:55] AlphaGo
https://arxiv.org/abs/1712.01815
[00:41:00] Open Source Generative AI
https://arxiv.org/abs/2405.08597
tool:
[00:09:45] JAX Library
https://github.com/jax-ml/jax
[00:49:51] Llama 3
https://ai.meta.com/blog/meta-llama-3/
concept:
[00:24:10] Goodharts Law
https://en.wikipedia.org/wiki/Goodhart%27s_law
LINKS:
Full Transcript: https://app.rescript.info/share/04498ba49b081dbcc254c40ba7b56035
Download PDF transcript: https://app.rescript.info/api/public/sessions/8fe06defb6d2216d/pdf
Prof. Jakob Foerster
https://x.com/j_foerst
https://www.jakobfoerster.com/
University of Oxford Profile:
https://eng.ox.ac.uk/people/jakob-foerster/
REFS
[[00:00:05] ARC Benchmark, Chollet
https://github.com/fchollet/ARC-AGI
[00:09:45] JAX Library, Google Research
https://github.com/jax-ml/jax
[00:25:15] LLM ARChitect, Franzen et al.
https://github.com/da-fr/arc-prize-2024/blob/main/the_architects.pdf")
![AI Isnt Creative [Prof. Kenneth Stanley]](https://i.ytimg.com/vi/KKUKikuV58o/mqdefault.jpg "AI Isnt Creative [Prof. Kenneth Stanley]
Are the AI models you use today imposters?
Please watch the intro video we did before this: https://www.youtube.com/watch?v=o1q6Hhz0MAg
In this episode, hosts Dr. Tim Scarfe and Dr. Duggar are joined by AI researcher Prof. Kenneth Stanley and MIT PhD student Akash Kumar to discuss their fascinating paper, Questioning Representational Optimism in Deep Learning.
Imagine you ask two people to draw a perfect skull. One is a brilliant artist who understands anatomy, the other is a machine that just traces the image. Both drawings look identical, but the artist understands what a skull is—they know where the mouth is, how the jaw works, and that its symmetrical. The machine just has a tangled mess of lines that happens to form the right picture.
An AI with an elegant representation, has the building blocks to generate truly new ideas.
The Path Is the Goal: As Kenneth Stanley puts it, it matters not just where you get, but how you got there. Two students can ace a math test, but the one who truly understands the concepts—instead of just memorizing formulas—is the one who will go on to make new discoveries.
The show is a mixture of 3 separate recordings we have done, the original Patreon warmup with Tim/Kenneth, the Tim/Keith Steakhouse recorded after the main interview, then the main interview with Kenneth/Akarsh/Keith/Tim. Feel free to skip around. We had to edit this in a rush as we are travelling next week but its reasonably cleaned up.
DUPLICATION NOTE:
There is a little bit of content duplication/overlap in these segments, if you are time-limited and dont want duplication - just watch the main interview from 48 mins
TOC:
00:00:00 Intro: Garbage vs. Amazing Representations
00:05:42 How Good Representations Form
00:11:14 Challenging the Bitter Lesson
00:18:04 AI Creativity & Representation Types
00:22:13 Steakhouse: Critiques & Alternatives
00:28:30 Steakhouse: Key Concepts & Goldilocks Zone
00:39:42 Steakhouse: A Sober View on AI Risk
00:43:46 Steakhouse: The Paradox of Open-Ended Search
00:47:58 Main Interview: Paper Intro & Core Concepts
00:56:44 Main Interview: Deception and Evolvability
01:36:30 Main Interview: Reinterpreting Evolution
01:56:16 Main Interview: Impostor Intelligence
02:11:15 Main Interview: Recommendations for AI Research
REFS:
Questioning Representational Optimism in Deep Learning:
The Fractured Entangled Representation Hypothesis
Akarsh Kumar, Jeff Clune, Joel Lehman, Kenneth O. Stanley
https://arxiv.org/pdf/2505.11581
Kenneth O. Stanley, Joel Lehman
Why Greatness Cannot Be Planned: The Myth of the Objective
https://amzn.to/44xLaXK
Original show with Kenneth from 4 years ago:
https://www.youtube.com/watch?v=lhYGXYeMq_E
Kenneth Stanley is SVP Open Endedness at Lila Sciences
https://x.com/kenneth0stanley
Akarsh Kumar (MIT)
https://akarshkumar.com/
AND... Kenneth is HIRING (this is an OPPORTUNITY OF A LIFETIME!)
Research Engineer: https://job-boards.greenhouse.io/lila/jobs/7890007002
Research Scientist: https://job-boards.greenhouse.io/lila/jobs/8012245002
TRANSCRIPT:
https://app.rescript.info/public/share/yVOfAlH9dwWoRozJGYrDiYSUlMFtRkzFPMW2S-9OQlI")
![The Real Reason Huge AI Models Actually Work [Prof. Andrew Wilson]](https://i.ytimg.com/vi/M-jTeBCEGHc/mqdefault.jpg "The Real Reason Huge AI Models Actually Work [Prof. Andrew Wilson]
Why can billion-parameter models perform so well without catastrophically overfitting? The answer lies in the mysterious simplicity bias that emerges at scale, a core concept of the double descent phenomenon.
Professor Andrew Wilson from NYU explains why many common-sense ideas in artificial intelligence might be wrong. For decades, the rule of thumb in machine learning has been to fear complexity. The thinking goes: if your model has too many parameters (is too complex) for the amount of data you have, it will overfit by essentially memorizing the data instead of learning the underlying patterns. This leads to poor performance on new, unseen data. This is known as the classic bias-variance trade-off i.e. a balancing act between a model thats too simple and one thats too complex.
**SPONSOR MESSAGES**
—
Tufa AI Labs is an AI research lab based in Zurich. **They are hiring ML research engineers!**
This is a once in a lifetime opportunity to work with one of the best labs in Europe
Contact Benjamin Crouzier - https://tufalabs.ai/
—
Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
Description Continued:
Professor Wilson challenges this fundamental belief (fearing complexity). He makes a few surprising points:
**Bigger Can Be Better**: massive models dont just get more flexible; they also develop a stronger simplicity bias. So, if your model is overfitting, the solution might paradoxically be to make it even bigger.
**The Bias-Variance Trade-off is a Misnomer**: Wilson claims you dont actually have to trade one for the other. You can have a model that is incredibly expressive and flexible while also being strongly biased toward simple solutions. He points to the double descent phenomenon, where performance first gets worse as models get more complex, but then surprisingly starts getting better again.
**Honest Beliefs and Bayesian Thinking**: His core philosophy is that we should build models that honestly represent our beliefs about the world. We believe the world is complex, so our models should be expressive. But we also believe in Occams razor—that the simplest explanation is often the best. He champions Bayesian methods, which naturally balance these two ideas through a process called marginalization, which he describes as an automatic Occams razor.
TOC:
[00:00:00] Introduction and Thesis
[00:04:19] Challenging Conventional Wisdom
[00:11:17] The Philosophy of a Scientist-Engineer
[00:16:47] Expressiveness, Overfitting, and Bias
[00:28:15] Understanding, Compression, and Kolmogorov Complexity
[01:05:06] The Surprising Power of Generalization
[01:13:21] The Elegance of Bayesian Inference
[01:33:02] The Geometry of Learning
[01:46:28] Practical Advice and The Future of AI
Prof. Andrew Gordon Wilson:
https://x.com/andrewgwils
https://cims.nyu.edu/~andrewgw/
https://scholar.google.com/citations?user=twWX2LIAAAAJ&hl=en
https://www.youtube.com/watch?v=Aja0kZeWRy4
https://www.youtube.com/watch?v=HEp4TOrkwV4
TRANSCRIPT:
https://app.rescript.info/public/share/H4Io1Y7Rr54MM05FuZgAv4yphoukCfkqokyzSYJwCK8
REFS:
Deep Learning is Not So Mysterious or Different [Andrew Gordon Wilson]
https://arxiv.org/abs/2503.02113
Bayesian Deep Learning and a Probabilistic Perspective of Generalization [Andrew Gordon Wilson, Pavel Izmailov]
https://arxiv.org/abs/2002.08791
Compute-Optimal LLMs Provably Generalize Better With Scale [Marc Finzi, Sanyam Kapoor, Diego Granziol, Anming Gu, Christopher De Sa, J. Zico Kolter, Andrew Gordon Wilson]
https://arxiv.org/abs/2504.15208")
