Uploaded November 2025 | Updated September 2026, 1 week ago
Tom Froese
oist.jp/research/research-units/ecsu/tom-froese
Will be out soon, on Patreon first
Machine Learning Street Talk
|
Is the Mind More Than Just the Brain? - Tom Froese
🔇 TAP TO UNMUTE
![The Weird ChatGPT Hack That Leaked Training Data [Dr. Yannic Kilcher / Prof. Florian Tramer]](https://i.ytimg.com/vi/c_hmxRVDXBE/mqdefault.jpg "The Weird ChatGPT Hack That Leaked Training Data [Dr. Yannic Kilcher / Prof. Florian Tramer]
Dr. Yannic lightspeed Kilcher sits down with an AI Security Researcher Prof. Florian Tramer (an expert in the field, collaborating with folks like Nicholas Carlini and Milad Nasr at Google) to ruminate on the murky state of affairs in AI security.
Out of the gate they assert that AI detectors (those tools that try to spot if text is human-written or AI-generated) are basically doomed to fail—like trying to catch an eel with greasy hands blindfolded. Lightspeed Kilcher shares a funny story about how the word Delve kept rearing its ugly head in AI text because of training data emanating from Nigerian crowd workers, leading to false flags in job applications.
Tramer then takes the gloves off about security risks, saying that even a 99% success rate in AI means total failure in security terms—attackers will always find that 1% weak spot. They share an interesting example of a weird attack on ChatGPT: asking it to repeat poem forever made it spit out memorized internet scraps, leaking training data like a sieve. OpenAI patched it, but its like slapping a band-aid on a shark bite.
Lightspeed Kilcher asks about top worries, and Tramer lists three big ones in relatable terms: 1) Training AI on private data (like medical records) could lead to massive leaks, since models memorize stuff without us fully understanding how; 2) Prompt injections, where bad actors hijack AI agents to do harmful things, like Anthropics demo that controls computers—its like giving your keys to a stranger; and 3) The rush to deploy AI without safeguards, echoing past tech blunders like SQL hacks.
They reflect on how ChatGPT has lit a fire under AI security research—making it amazing and scary by turning hypotheticals into real-world issues with millions of users. The conversation touches on why just scaling up AI wont fix everything (we need causal understanding, not just more data), and why watermarking AI outputs isnt a silver bullet—its easy to bypass, especially with open-source models.
Its about time we had our old comrade light speed Kilcher back on MLST!
TRANSCRIPT:
https://app.rescript.info/public/share/PGieXFZCRyJjJswicatmNFia_pKCt_WVP-pZS4qguiw
LINKS:
Scalable Extraction of Training Data from (Production) Language Models [Poem Attack]
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel
https://arxiv.org/abs/2311.17035
- Prof. Florian Tramer
https://www.floriantramer.com/
- Yannic Kilchers YouTube channel
https://www.youtube.com/c/YannicKilcher
- Nicholas Carlinis research profile (collaborator on the data recovery project):
https://nicholas.carlini.com/ – Includes papers on AI attacks and memorization risks.")
![AI training data will never be fully synthetic [SPONSORED]](https://i.ytimg.com/vi/cnxZZTl1tkk/mqdefault.jpg "AI training data will never be fully synthetic [SPONSORED]
We sat down with Sara Saab (VP of Product at Prolific) and Enzo Blindow (VP of Data and AI at Prolific) to explore the critical role of human evaluation in AI development and the challenges of aligning AI systems with human values. Prolific is a human annotation and orchestration platform for AI used by many of the major AI labs. This is a sponsored show in partnership with Prolific.
**SPONSOR MESSAGES**
—
cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy
Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++
Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst
Submit investment deck: https://cyber.fund/contact?utm_source=mlst
—
While technologists want to remove humans from the loop for speed and efficiency, these non-deterministic AI systems actually require more human oversight than ever before. Prolifics approach is to put well-treated, verified, diversely demographic humans behind an API - making human feedback as accessible as any other infrastructure service.
When AI models like Grok 4 achieve top scores on technical benchmarks but feel awkward or problematic to use in practice, it exposes the limitations of our current evaluation methods. The guests argue that optimizing for benchmarks may actually weaken model performance in other crucial areas, like cultural sensitivity or natural conversation.
We also discuss Anthropics research showing that frontier AI models, when given goals and access to information, independently arrived at solutions involving blackmail - without any prompting toward unethical behavior. Even more concerning, the more sophisticated the model, the more susceptible it was to this agentic misalignment.
Enzo and Sarah present Prolifics Humane leaderboard as an alternative to existing benchmarking systems. By stratifying evaluations across diverse demographic groups, they reveal that different populations have vastly different experiences with the same AI models.
Looking forwards the guests imagine a world where humans take on coaching and teaching roles for AI systems - similar to how we might correct a child or review code. Working conditions and the evolution of labor in an AI-augmented world is also important to consider. Rather than replacing humans entirely, we may be moving toward more sophisticated forms of human-AI collaboration.
We need more representative evaluation frameworks that capture the messy reality of human values and cultural diversity.
Visit Prolific:
https://www.prolific.com/
Sara Saab (VP Product):
https://uk.linkedin.com/in/sarasaab
Enzo Blindow (VP Data & AI):
https://uk.linkedin.com/in/enzoblindow
TRANSCRIPT:
https://app.rescript.info/public/share/xZ31-0kJJ_xp4zFSC-bunC8-hJNkHpbm7Lg88RFcuLE
TOC:
[00:00:00] Intro & Background
[00:03:16] Human-in-the-Loop Challenges
[00:17:19] Can AIs Understand?
[00:32:02] Benchmarking & Vibes
[00:51:00] Agentic Misalignment Study
[01:03:00] Data Quality vs Quantity
[01:16:00] Future of AI Oversight
REFS:
Anthropic Agentic Misalignment
https://www.anthropic.com/research/agentic-misalignment
Value Compass
https://arxiv.org/pdf/2409.09586
Reasoning Models Don’t Always Say What They Think (Anthropic)
https://www.anthropic.com/research/reasoning-models-dont-say-think
https://assets.anthropic.com/m/71876fabef0f0ed4/original/reasoning_models_paper.pdf
Maslow’s Hierarchy Of Needs
https://www.simplypsychology.org/maslow.html
Apollo research - science of evals blog post
https://www.apolloresearch.ai/blog/we-need-a-science-of-evals
Leaderboard Illusion
https://www.youtube.com/watch?v=9W_OhS38rIE MLST video
The Leaderboard Illusion [2025]
Shivalika Singh, Yiyang Nan, Alex Wang, Daniel DSouza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah A. Smith, Beyza Ermis, Marzieh Fadaee, Sara Hooker
https://arxiv.org/abs/2504.20879
Humanities last exam
https://arxiv.org/abs/2501.14249
PRISM paper
https://arxiv.org/abs/2405.10254
https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback
Collective intelligence project
https://www.cip.org/whitepaper
https://ischool.utoronto.ca/faculty-profile/brian-cantwell-smith/
Ghost work (Mary Gray)
https://www.amazon.com/Ghost-Work-Silicon-Building-Underclass/dp/1328566242
Fairwork Cloudwork report
https://fair.work/en/ratings/cloudwork/
Gibson theory of affordances
https://cs.brown.edu/courses/cs137/2017/readings/Gibson-AFF.pdf")
![We Built Calculators Because Were STUPID! [Prof. David Krakauer]](https://i.ytimg.com/vi/dY46YsGWMIc/mqdefault.jpg "We Built Calculators Because Were STUPID! [Prof. David Krakauer]
Prof. David Krakauer argues that intelligence isnt about knowing more—its about doing more with less.
His core thesis flips our assumptions on their head: while emergence in complex systems is about more is different, intelligence is fundamentally about less is more.
Why Large Language Models Miss the Point
LLMs are more is more, Krakauer explains. Theyre essentially sophisticated libraries that know everything but understand nothing. When a student looks up answers in a library during an exam, we call them a cheater, not intelligent.
Intelligence Across All Life
Perhaps most surprisingly, Krakauer claims that all living things are intelligent—even bacteria. His reasoning: intelligence is the information accumulated through evolutions selective pressures. Its why we can say an elephant is smarter than a worm, but wed never say its more alive. The difference is accumulated capacity.
Three Dimensions of Intelligence
Krakauer proposes intelligence exists in three distinct flavours:
- Strategic Intelligence - Adaptation and survival where viruses actually beat us/
- Inferential Intelligence - Math and computation (where we built calculators because were so bad at it)
- Representational Intelligence - Finding better ways to encode problems (the most uniquely human)
The Soma Cube Insight
Using a Soma Cube puzzle, Krakauer demonstrates something incredible: a four-year-old can solve a combinatorial problem with 15,000 logical clauses—not by computing, but by using physical representation. The physical world does the computing for us. This is intelligence: making impossibly hard problems tractable through clever representation.
This talk was recorded at DISI 2025 https://disi.org/
David Krakauer
President and William H. Miller Professor of Complex Systems
https://www.santafe.edu/people/profile/david-krakauer
Note this version is edited down, you can watch the full version on our Patreon - https://www.patreon.com/posts/masterclass-on-142898847
Watch interview we published with David - https://www.youtube.com/watch?v=jXa8dHzgV8U
TRANSCRIPT:
https://app.rescript.info/public/share/Cs8QFwMw_FiOEROkNPiIf6Y-e_ahAxNMl0oFiL7_4QA
TOC:
00:00:00 - Complexity, Life & Intelligence
00:00:45 - Ptolemy vs Newton
00:01:30 - Purpose of Science
00:02:15 - Feynman on Rules vs Strategies
00:03:00 - Entropy as Unifying Rule
00:03:45 - Intelligence in Physics
00:04:15 - Complex Systems & Broken Symmetries
00:05:00 - Historical Definitions of Intelligence
00:06:15 - Herbert: Capacity to Acquire Capacity
00:06:45 - Problems with Turing Test
00:07:30 - Intelligence vs Knowledge
00:08:00 - LLMs as Libraries
00:08:30 - Humanistic Perspectives
00:09:00 - Unified Theory: Life & Intelligence
00:09:30 - Intensive vs Extensive Properties
00:10:30 - Selection Gradient Analogy
00:11:15 - Universal Intelligence (Bacteria to Elephants)
00:12:30 - Three Dimensions of Intelligence
00:13:00 - Strategic Intelligence & Viruses
00:13:30 - Inferential Intelligence & Calculators
00:14:15 - Encoding & Representation
00:14:30 - Different Paths: Life vs AI
00:15:15 - Soma Cube Problem
00:16:00 - Principle of Materiality
00:17:15 - Embodied Representation
00:17:45 - Intelligence Makes Hard Problems Easy
00:18:00 - Stupidity & Conclusion
REFS:
Papers and Academic Works
More is Different by Phil Anderson [https://www.tkm.kit.edu/downloads/TKM1_2011_more_is_different_PWA.pdf ]
Recent paper with Melanie and John Krakauer on emergence [https://arxiv.org/abs/2506.11135 ]
Formal treatment paper on intelligence/evolution duality from David - Darwinian demons, evolutionary complexity, and information maximization [https://wiki.santafe.edu/images/b/b1/CHAOEH213037110_1.pdf ]
The Unreasonable Effectiveness of Mathematics in the Natural Sciences by Eugene Wigner [https://webhomes.maths.ed.ac.uk/~v1ranick/papers/wigner.pdf ]")
