Leonard Dung - Is Superintelligence Necessarily Moral? @scfu
Leonard Dung - Is Superintelligence Necessarily Moral?  @scfu
Uploaded September 2026 | Updated November 2026, 15 hours ago
Can a superintelligent AI understand morality without being motivated by it?

I speak with philosopher Leonard Dung about his paper Is Superintelligence Necessarily Moral?, which examines Nick Bostrom’s orthogonality thesis and challenges arguments that sufficiently intelligent AI systems would necessarily acquire morally good goals.

We discuss the distinction between moral cognition and moral motivation; moral realism and motivational internalism; whether intelligence and values can really vary independently; Peter Railton’s challenge to orthogonality; reinforcement learning and reward hacking; the Obliqueness Thesis; instrumental convergence; alignment faking; emergent values in language models; and what evidence might show that increasingly capable AI systems are becoming wiser rather than merely better at talking about morality.

We also discuss how much present-day AI can tell us about future superintelligence, whether reward functions should really be understood as an AI system’s “final goals”, and what a convincing empirical research programme on AI moral motivation might look like.

Leonard Dung is a philosopher at Ruhr University Bochum whose research includes AI alignment, existential risk, AI sentience and moral status. He is also the author of Saving Artificial Minds: Understanding and Preventing AI Suffering.

Topics include: AI alignment, orthogonality, superintelligence, moral realism, moral motivation, motivational internalism, reward hacking, instrumental convergence, AI values and machine ethics.

#AIAlignment #AISafety #Superintelligence #MoralPhilosophy #AIEthics

Dr. Leonard Dung is a Postdoctoral Researcher at the Institute of Philosophy II, Ruhr-University Bochum

- What is the core argument of "Is Superintelligence Necessarily Moral?"
- Can moral knowledge motivate artificial agents?
- How does this argument relate to Bostrom’s orthogonality thesis?
- Does moral realism change the alignment problem?
- What do current AI systems suggest about the relationship between learning, goals, and motivation?

See paper 'Is superintelligence necessarily moral?': philarchive.org/rec/DUNISN

Also see paper 'Saving Artificial Minds Understanding and Preventing AI Suffering': routledge.com/Saving-Artificial-Minds-Understanding-and-Preventing-AI-Suffering/Dung/p/book/9781041144663

And SciFuture blog post: scifuture.org/is-superintelligence-necessarily-moral-discussion-with-leonard-dung

Chapters:

0:00 Introduction – Leonard Dung
1:26 Is Superintelligence Necessarily Moral?
2:59 What exactly does the Orthogonality Thesis claim?
5:49 Orthogonality: possibility vs prediction
9:59 Moral realism and what counts as a “good” AI
15:08 Moral reasoning vs moral motivation
20:45 Motivational internalism and AI
24:47 Could AI learn morality rather than have values programmed?
29:32 Peter Railton: are intelligence and morality entangled?
35:45 What evidence could link reasoning and motivation?
39:43 How could we experimentally test AI moral motivation?
43:16 Goal preservation as intelligence increases
46:42 The Obliqueness Thesis vs Orthogonality
50:03 Weak vs strong versions of Orthogonality
52:02 AI timelines and possible warning signs
54:08 Human misuse vs intrinsically misaligned AI
57:58 Can moral motivation be empirically tested?
1:00:36 Reward maximisation, deception and learned motivations
1:04:23 Simulating dangerous AI behaviour
1:06:28 What would successful alignment look like?
1:08:47 Interpretability and understanding AI motivations
1:12:02 Claude and alignment faking
1:14:00 Whose values should AI be aligned to?
1:15:59 What causes alignment faking?
1:18:56 How much evidence is enough to trust superintelligence?
1:22:54 Could today's AI improve public reasoning?
1:24:48 Reward signals vs emergent AI motivations
1:29:39 Has frontier AI changed Leonard’s views?
1:33:28 Should we pursue superintelligence despite the risks?
1:34:46 Suffering risks and emergent misalignment
1:36:29 Final takeaways

Many thanks for tuning in!
Please support SciFuture by subscribing and sharing!
Buy me a coffee? buymeacoffee.com/tech101z

Have any ideas about people to interview? Want to be notified about future events? Any comments about the STF series?
Please fill out this form: docs.google.com/forms/d/1mr9PIfq2ZYlQsXRIn5BcLH2onbiSI7g79mOH_AFCdIk

Kind regards,
Adam Ford
- Science, Technology & the Future - #SciFuture - scifuture.org
Leonard Dung - Is Superintelligence Necessarily Moral?Adam Ford - AI Convergence on Coherent Value Systems - Future Day LivestreamRoman Yampolskiy – AI: Unexplainable, Unpredictable, UncontrollableRoman Yampolskiy - AI: the Double Edged SwordTerence Cuneo: Epistemic Nihilism - Armageddon for Rational Agency #ethics #metaethics #nihilism
Science, Technology & the Future |

Leonard Dung - Is Superintelligence Necessarily Moral?

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER