Uploaded February 2025 | Updated September 2026, 2 weeks ago
Can #llms truly reason, or are they just repeating patterns?
In this talk, Andrew Lampinen from @googledeepmind explores how cognitive science provides useful tools to analyze #LLM capabilities. He discusses comparative methods (comparing different systems) and rational analysis (examining behaviors as rational adaptations). Andrew also shares his research on how both LLMs and humans entangle content in logical #reasoning problems and the broader implications for improving #ai performance.
Timestamps:
0:00 Introduction
1:10 Why is it useful to apply cognitive methods to analyze language models capabilities
3:15 How do we "motivate" the model to solve the problem?
3:53 Logical reasoning: What is reasoning? What is human reasoning?
5:49 People judge logical arguments by their content, not just by logical structure
7:21 Content can facilitate rule reasoning
8:53 There is significant interest in LLM reasoning
9:25 Why be skeptical about reasoning in LLMs?
10:11 Three datasets for evaluating content effects
11:53 LLMs vs. Humans: performance analysis of an experiment
16:49 Evaluate humans on tasks if you want to compare them to models
17:42 Reasoning task performance is multifactorial for both humans and LLMs
20:27 To encourage abstract reasoning, shift the training distribution to make it optimal
#artificialintelligence #largelanguagemodels #llms #aireasoning #cognitivescience #science #aimodel #machinelearning #reinforcementlearning #airesearch #aichallenges #techtalk #techtalks #aitalks #aitalk #science #python #pythonprogrammingcourse #ai #programming
Social Links:
Newsletter: buzzrobot.substack.com
X: https://x.com/sopharicks
Slack: join.slack.com/t/buzzrobot/shared_invite/zt-2s067rv7n-guPIMGe62rbp9ncxdnOUfQ
Can #llms truly reason, or are they just repeating patterns?
In this talk, Andrew Lampinen from @googledeepmind explores how cognitive science provides useful tools to analyze #LLM capabilities. He discusses comparative methods (comparing different systems) and rational analysis (examining behaviors as rational adaptations). Andrew also shares his research on how both LLMs and humans entangle content in logical #reasoning problems and the broader implications for improving #ai performance.
Timestamps:
0:00 Introduction
1:10 Why is it useful to apply cognitive methods to analyze language models capabilities
3:15 How do we "motivate" the model to solve the problem?
3:53 Logical reasoning: What is reasoning? What is human reasoning?
5:49 People judge logical arguments by their content, not just by logical structure
7:21 Content can facilitate rule reasoning
8:53 There is significant interest in LLM reasoning
9:25 Why be skeptical about reasoning in LLMs?
10:11 Three datasets for evaluating content effects
11:53 LLMs vs. Humans: performance analysis of an experiment
16:49 Evaluate humans on tasks if you want to compare them to models
17:42 Reasoning task performance is multifactorial for both humans and LLMs
20:27 To encourage abstract reasoning, shift the training distribution to make it optimal
#artificialintelligence #largelanguagemodels #llms #aireasoning #cognitivescience #science #aimodel #machinelearning #reinforcementlearning #airesearch #aichallenges #techtalk #techtalks #aitalks #aitalk #science #python #pythonprogrammingcourse #ai #programming
Social Links:
Newsletter: buzzrobot.substack.com
X: https://x.com/sopharicks
Slack: join.slack.com/t/buzzrobot/shared_invite/zt-2s067rv7n-guPIMGe62rbp9ncxdnOUfQ
