Uploaded February 2025 | Updated September 2026, 1 week ago
Machine Learning Street Talk
|
Not real reasoning?
🔇 TAP TO UNMUTE
![The ARC Prize 2024 Winning Algorithm [Daniel Franzen and Jan Disselhoff]](https://i.ytimg.com/vi/mTX_sAq--zY/mqdefault.jpg "The ARC Prize 2024 Winning Algorithm [Daniel Franzen and Jan Disselhoff]
SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!
https://centml.ai/pricing/
Daniel Franzen and Jan Disselhoff the ARChitects won the ARC Prize 2024 with co-researcher David Hartmann, achieving a remarkable 53.5% accuracy on the ARC challenge using a 12-billion parameter language model. Filmed at Tufa Labs in Zurich, they walk through their solution architecture in detail: how they tokenized grid-based visual puzzles as text and fed them directly into an LLM, why test-time training on evaluation examples gave a major score boost, and how depth-first search over token probabilities outperformed greedy and beam search for solution generation.
The conversation gets into the surprising computational capabilities of language models on spatial reasoning tasks. The team found that LLMs could handle 2D grid reasoning without explicit positional encodings, that symmetry augmentations served as a powerful validation mechanism rather than a training aid, and that the models second-best solutions were often conceptually correct just wrong in a specific detail like rotation direction. They also discuss why their fine-tuned 12B model outperformed much larger foundation models, the role of LoRA in preventing catastrophic forgetting during test-time training, and what the entropy distribution of their search trees reveals about how the model represents uncertainty across different task types.
REFERENCES:
Paper:
[00:01:00] The ARChitects: Winning ARC Prize 2024 Solution
https://github.com/da-fr/arc-prize-2024/blob/main/the_architects.pdf
[00:03:38] Robustness of Analogical Reasoning in LLMs
https://arxiv.org/html/2411.14215
[00:14:58] Search Methods in Language Models
https://arxiv.org/html/2408.00724v2
[00:22:28] GPT-4 Code Solutions for ARC (50% SOTA)
https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt
[00:53:08] Overcoming Catastrophic Forgetting
https://www.pnas.org/doi/10.1073/pnas.1611835114
[00:53:58] LoRA: Low-Rank Adaptation of Large Language Models
https://arxiv.org/abs/2106.09685
Tool:
[00:07:48] Re-ARC Dataset Generator
https://github.com/michaelhodel/re-arc
LINKS:
Full Transcript: https://app.rescript.info/share/57e5d773f2d0b195cbce7eee1f53aef2
Download PDF transcript: https://app.rescript.info/api/public/sessions/7772acbf1f11f44b/pdf
Daniel Franzen
https://github.com/da-fr
REFS
[00:01:05] Winning ARC 2024 solution using 12B param model, Franzen, Disselhoff, Hartmann
https://github.com/da-fr/arc-prize-2024/blob/main/the_architects.pdf
[00:07:50] Re-ARC dataset generator for ARC task variations, Michael Hodel
https://github.com/michaelhodel/re-arc
[00:22:30] GPT-4 guided code solutions for ARC tasks, Ryan Greenblatt
https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt")
![Language Models are Modelling The World [Nicholas Carlini]](https://i.ytimg.com/vi/n4ipEJ6uJ44/mqdefault.jpg "Language Models are Modelling The World [Nicholas Carlini]
Nicholas Carlini from Google DeepMind offers his view of AI security, emergent LLM capabilities, and his groundbreaking model-stealing research. He reveals how LLMs can unexpectedly excel at tasks like chess and discusses the security pitfalls of LLM-generated code.
SPONSOR MESSAGES:
***
CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.
https://centml.ai/pricing/
Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events?
Goto https://tufalabs.ai/
***
Transcript: https://www.dropbox.com/scl/fi/lat7sfyd4k3g5k9crjpbf/CARLINI.pdf?rlkey=b7kcqbvau17uw6rksbr8ccd8v&dl=0
TOC:
1. ML Security Fundamentals
[00:00:00] 1.1 ML Model Reasoning and Security Fundamentals
[00:03:04] 1.2 ML Security Vulnerabilities and System Design
[00:08:22] 1.3 LLM Chess Capabilities and Emergent Behavior
[00:13:20] 1.4 Model Training, RLHF, and Calibration Effects
2. Model Evaluation and Research Methods
[00:19:40] 2.1 Model Reasoning and Evaluation Metrics
[00:24:37] 2.2 Security Research Philosophy and Methodology
[00:27:50] 2.3 Security Disclosure Norms and Community Differences
3. LLM Applications and Best Practices
[00:44:29] 3.1 Practical LLM Applications and Productivity Gains
[00:49:51] 3.2 Effective LLM Usage and Prompting Strategies
[00:53:03] 3.3 Security Vulnerabilities in LLM-Generated Code
4. Advanced LLM Research and Architecture
[00:59:13] 4.1 LLM Code Generation Performance and O(1) Labs Experience
[01:03:31] 4.2 Adaptation Patterns and Benchmarking Challenges
[01:10:10] 4.3 Model Stealing Research and Production LLM Architecture Extraction
REFS:
[00:01:15] Nicholas Carlini’s personal website & research profile (Google DeepMind, ML security) - https://nicholas.carlini.com/
[00:01:50] CentML AI compute platform for language model workloads - https://centml.ai/
[00:04:30] Seminal paper on neural network robustness against adversarial examples (Carlini & Wagner, 2016) - https://arxiv.org/abs/1608.04644
[00:05:20] Computer Fraud and Abuse Act (CFAA) – primary U.S. federal law on computer hacking liability - https://www.justice.gov/jm/jm-9-48000-computer-fraud
[00:08:30] Blog post: Emergent chess capabilities in GPT-3.5-turbo-instruct (Nicholas Carlini, Sept 2023) - https://nicholas.carlini.com/writing/2023/chess-llm.html
[00:16:10] Paper: “Self-Play Preference Optimization for Language Model Alignment” (Yue Wu et al., 2024) - https://arxiv.org/abs/2405.00675
[00:18:00] GPT-4 Technical Report: development, capabilities, and calibration analysis - https://arxiv.org/abs/2303.08774
[00:22:40] Historical shift from descriptive to algebraic chess notation (FIDE) - https://en.wikipedia.org/wiki/Descriptive_notation
[00:23:55] Analysis of distribution shift in ML (Hendrycks et al.) - https://arxiv.org/abs/2006.16241
[00:27:40] Nicholas Carlini’s essay “Why I Attack” (June 2024) – motivations for security research - https://nicholas.carlini.com/writing/2024/why-i-attack.html
[00:34:05] Google Project Zero’s 90-day vulnerability disclosure policy - https://googleprojectzero.blogspot.com/p/vulnerability-disclosure-policy.html
[00:51:15] Evolution of Google search syntax & user behavior (Daniel M. Russell) - https://www.amazon.com/Joy-Search-Google-Master-Information/dp/0262042878
[01:04:05] Rust’s ownership & borrowing system for memory safety - https://doc.rust-lang.org/book/ch04-00-understanding-ownership.html
[01:10:05] Paper: “Stealing Part of a Production Language Model” (Carlini et al., March 2024) – extraction attacks on ChatGPT, PaLM-2 - https://arxiv.org/abs/2403.06634
[01:10:55] First model stealing paper (Tramèr et al., 2016) – attacking ML APIs via prediction - https://arxiv.org/abs/1609.02943")
