Metacognition in LLMs: Can AI Think About Thinking? - Shun Yoshizawa & Ken Mogi @scfu
Metacognition in LLMs: Can AI Think About Thinking? - Shun Yoshizawa & Ken Mogi  @scfu
Uploaded May 2025 | Updated November 2026, 1 day ago
Do LLMs have metacognition?
The Future Day talk 'Metacognition in LLMs' with Shun Yoshizawa & Ken Mogi explores this question, the fact that #LLMs are often overconfident, and implications for robust #metacognition in #AI.

Accompanying article: scifuture.org/metacognition-in-large-language-models

ChatGPT has shown robust performance in false belief tasks, suggesting it has a theory of mind. It might be important to assess how accurately LLMs can be aware of their own performances. Here we investigate the general metacognitive abilities of the LLMs by analysing LLM and humans confidence judgements. Human subjects tended to be less confident when they answered incorrectly than when they answered correctly. However, the GPT-4 showed high confidence even in the questions that they could not answer correctly. These results suggest that GPT-4 lacks specific metacognitive abilities.

0:00:00 Introduction and Speaker Introduction
0:00:58 Metacognition in Large Language Models
0:02:50 Review of Serial Mind in LLMs
0:04:10 LLMs as Cognitive Models: Limitations
0:05:38 Defining Metacognition
0:06:28 Conventional Metacognition Studies
0:07:51 Research Findings: GPT-4 Metacognition
0:12:08 Study Conclusions and Implications
0:13:13 Application to AI Alignment and Hallucination
0:14:16 Metacognition Increases with Model Capability
0:14:52 Deceptive Misalignment Types
0:15:58 Future Metacognition Research Directions
0:16:21 Inference Time and Epistemic Humility
0:17:35 Discussion Opens to Q&A
0:18:14 Audience Question: Prompt Engineering
0:20:09 Prompting Conditions vs. Typical Use
0:21:50 Mechanistic Interpretability for Humility
0:23:15 AI Self-Introspection
0:24:47 Human vs. AI Access to Internal States
0:26:02 Can AI Red Team Itself?
0:27:16 AI Safety Trade-offs
0:28:48 Instilling AI Principles: Humility, Care
0:29:46 AGI/ASI and Humility
0:30:35 Overconfidence, Value Change in AI
0:31:19 Embedding Metacognition in LLMs
0:31:44 Artificial General Competence (AGC)
0:32:43 Crucial Moments for Metacognition Testing
0:34:26 LLM Memory and Internal State
0:35:48 Metacognitive Prompting and RAGs
0:36:46 Kernel vs. Module Approach
0:37:19 Exciting Metacognition Research
0:38:22 Game Theory and the Language Game
0:40:40 Defining and Grounding the Language Game
0:42:14 Meaning, Use, and Absurdity in Language
0:43:16 Conversational Maxims and Implicature
0:44:30 Distributional Semantics and Wittgenstein
0:45:27 Wittgenstein's Relevance to AI
0:46:05 Metacognition, Consciousness, and AI
0:47:50 Metacognition and Deceptive AI Behavior
0:49:37 Intentional Alignment
0:50:25 Infinite Regression and AI Challenges
0:50:55 Hallucinations, Overconfidence, Maximization
0:51:44 Mitigating AI Failures via Metacognition
0:53:24 Halting Problem and Resource Management
0:54:37 Spectrum of Metacognition and Looping
0:55:38 Embodiment and Stopping Computation
0:57:10 Reinforcement Learning and Metacognition
0:57:49 Emergent Metacognition Possibility
0:59:03 Turing Test and Behavioral Criteria
0:59:38 Social Adaptation and AI Metacognition
1:01:53 Training AI for Metacognitive Skills
1:03:02 AI Safety, Ethics, and Accountability
1:03:45 Discussion on the Halting Problem
1:06:46 Post-Singularity in Go and Science
1:07:47 Understanding Superhuman AI Solutions
1:08:30 Compression and Human Understanding of AI
1:09:36 Metacognition as Compressed Representation
1:10:03 Understanding Requires Metacognition
1:10:34 Machine Understanding Research
1:11:36 Emergent Capabilities Beyond Parroting
1:12:11 Singularity Timelines
1:12:53 Measuring Understanding and Metacognition
1:13:38 The Opaque Box Problem
1:14:07 Types of Understanding and AI
1:14:36 Measuring Superhuman Intelligence
1:15:06 Verifiable Rewards and Scientific Discovery
1:16:33 Designing Complex Rewards for AI
1:17:41 AI for Ethics and Human Flourishing
1:18:15 High Dimensionality Challenges
1:19:38 Visualizing High Dimensions
1:20:16 The Excitement of Being Alive Now
1:20:51 Optimism vs. Doomerism in AI
1:21:32 AI Doomerism and Resolution
1:21:59 Approaches to AI Safety: Mathematical Precision
1:23:17 Swiss Cheese Approach to AI Safety
1:24:49 Multiple Layers of Defense
1:25:02 Optimism and Curiosity
1:25:43 AI Assisting in AI Safety
1:26:07 Indirect Normativity
1:27:16 Coherent Extrapolated Volition (CEV)
1:29:33 Critiques and Refining Ethics
1:30:43 Dystopia/Utopia Closeness and Leibniz
1:32:01 Leibniz and AI Alignment
1:33:05 Roman Yampolskiy and Doomerism
1:33:56 AI, Moloch and Race to the Bottom
1:34:16 Closing Remarks and Appreciation

#Metacognition #ai #llm

Please support SciFuture by subscribing and sharing!
Buy me a coffee? buymeacoffee.com/tech101z

Kind regards,
Adam Ford
- Science, Technology & the Future - #SciFuture - scifuture.org
Metacognition in LLMs: Can AI Think About Thinking? - Shun Yoshizawa & Ken MogiTerence Cuneo: Moral Facts - Grounding Non-Moral Facts #ethics #philosophyJoscha Bach: Galactic Game Theory - AI Convergence #AI #ASI #GameTheoryTerence Cuneo: AI Alignment - Navigating Human ValuesJoscha Bach & Anders Sandberg - AI, Consciousness and the Cyborg LeviathanTerence Cuneo - Exploring the Parallels Between #Moral and Epistemic Facts #ethics #philosophyPhan Xuan Tan - AI Safety in Education: Preparing the Next Generation for Responsible AI DevelopmentIs Superintelligence Necessarily Moral?Are Machines Capable of Morality? Join Professor Colin Allen!Soenke Ziesche - IkigAI & AI, Ikigai Risk (IRisk)Wendell Wallach: AI Ethics in China - a Governance Mirror #aigovernance #aiethics  #aialignmentAngela Livingstone - AI Won’t Replace Psychiatrists - But May Expose Us
Science, Technology & the Future |

Metacognition in LLMs: Can AI Think About Thinking? - Shun Yoshizawa & Ken Mogi

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER