Uploaded June 2025 | Updated November 2026, 15 hours ago
Since Nick Bostrom wrote Superintelligence, AI has surged from theoretical speculation to powerful, world-shaping reality. Progress is undeniable, yet there is an ongoing debate in the AI safety community – caught between mathematical rigour and swiss-cheese security. P(doom) debates rage on, but equally concerning is the risk of locking in negative-value futures for a very long time.
Zooming in: motivation selection-especially indirect normativity-raises the question: is there a structured landscape of possible value configurations, or just a chaotic search for alignment?
From Superintelligence to Deep Utopia: not just avoiding catastrophe but ensuring resilience, meaning, and flourishing in a 'solved' world; a post instrumental, plastic utopia – where humans are 'deeply redundant', can we find enduring meaning and purpose?
This is our moment to shape the future. What values will we encode? What futures will we entrench?
0:00 Highlights
1:32 Intro
2:40 Interview - shifts in discourse
6:04 p(doom)? Likelihood that AI will doom us? What counts as doom?
7:34 From dismissal to acknowledgement - geopolitical dynamics surrounding AI advancements and safety measures
9:10 How soon superintelligence?
12:34 AI risk challenges: alignment, political challenges, ethical considerations, and fitting into broader cosmic context (i.e. a society of mature civs/cosmic hosts)
15:06 Cosmic hosts, trust, game theory & utility maximisation: the need for humility in deciding which norms to load into superintelligence
17:55 The dynamics of the landscape of value & initial trajectories effecting where superintelligence ends up
22:19 Maximisation tempered with humility, and distinguishing good from bad values - dichotomy between moral realism and anti-realism
23:55 Satiability vs insatiability
26:43 Metacognition and it's location in the landscape of value
29:12 AI alignment - control vs motivation
33:34 The Indirect normativity approach to AI alignment (alternative to capability control)
34:01 Elios AI
35:00 Indirect normativity to debug metaphysical, epistemic, metaethical assumptions
40:37 Indirect normativity in practise given the kind of AI we see today
42:58 AI safety: Swiss cheese vs mathematical precision
45:57 Do we need to provide AI with metaphysical/epistemic/metaethical axioms?
50:54 Epistemic deference to AI on concepts too complex for the human mind to grasp
53:29 Does AI understand stuff?
54:56 Human-like AI now b/c training data - perhaps alien-like AI soon based on new training regimes
57:30 Base Camp at Mt Ethics, cosmic norms as metaethics developed by a cosmic host
59:39 AI Creation & the Cosmic Host - train AI to defer to cosmic norms
1:03:20 Values as causal influencers of goals, so better get the values right
1:06:36 Deep Utopia if we get superintelligence right - what is Utopia
1:13:16 Shallow redundancy vs deep redundancy - subtle values coming into focus
1:14:12 Deep utopia may involve giving up some things. Workng through repulsiveness of utopain visions
1:14:39 Going back to basic constituents of values, from there rebuilding value
1:15:36 Philosophical particle accelerator
1:19:12 Taking the scenic route to posthuman forms instead of rushing
1:28:01 Hedonics and the 5 Ringed Defence of Meaning in a plastic utopia
1:40:20 The space of subtle values is enormous, Ikagai and Japanese culture
1:49:45 Maximum novelty experienced early in life?
1:52:04 Putting the picture together
1:55:34 Outriders, and Finding Purpose Now
1:57:04 Reflections on the old era of futurology - Conclusion
Elios AI eleosai.org
#ai #utopia #superintelligence #agi
The music at the end is 'Hedonium 1' which is guitar saturated with Strymon reverbs, delays and modulation: soundcloud.com/limping-fiddler/hedonium-1
Many thanks for tuning in!
Please support SciFuture by subscribing and sharing!
Buy me a coffee? buymeacoffee.com/tech101z
Have any ideas about people to interview? Want to be notified about future events? Any comments about the STF series?
Please fill out this form: docs.google.com/forms/d/1mr9PIfq2ZYlQsXRIn5BcLH2onbiSI7g79mOH_AFCdIk
Kind regards,
Adam Ford
- Science, Technology & the Future - #SciFuture - scifuture.org
Since Nick Bostrom wrote Superintelligence, AI has surged from theoretical speculation to powerful, world-shaping reality. Progress is undeniable, yet there is an ongoing debate in the AI safety community – caught between mathematical rigour and swiss-cheese security. P(doom) debates rage on, but equally concerning is the risk of locking in negative-value futures for a very long time.
Zooming in: motivation selection-especially indirect normativity-raises the question: is there a structured landscape of possible value configurations, or just a chaotic search for alignment?
From Superintelligence to Deep Utopia: not just avoiding catastrophe but ensuring resilience, meaning, and flourishing in a 'solved' world; a post instrumental, plastic utopia – where humans are 'deeply redundant', can we find enduring meaning and purpose?
This is our moment to shape the future. What values will we encode? What futures will we entrench?
0:00 Highlights
1:32 Intro
2:40 Interview - shifts in discourse
6:04 p(doom)? Likelihood that AI will doom us? What counts as doom?
7:34 From dismissal to acknowledgement - geopolitical dynamics surrounding AI advancements and safety measures
9:10 How soon superintelligence?
12:34 AI risk challenges: alignment, political challenges, ethical considerations, and fitting into broader cosmic context (i.e. a society of mature civs/cosmic hosts)
15:06 Cosmic hosts, trust, game theory & utility maximisation: the need for humility in deciding which norms to load into superintelligence
17:55 The dynamics of the landscape of value & initial trajectories effecting where superintelligence ends up
22:19 Maximisation tempered with humility, and distinguishing good from bad values - dichotomy between moral realism and anti-realism
23:55 Satiability vs insatiability
26:43 Metacognition and it's location in the landscape of value
29:12 AI alignment - control vs motivation
33:34 The Indirect normativity approach to AI alignment (alternative to capability control)
34:01 Elios AI
35:00 Indirect normativity to debug metaphysical, epistemic, metaethical assumptions
40:37 Indirect normativity in practise given the kind of AI we see today
42:58 AI safety: Swiss cheese vs mathematical precision
45:57 Do we need to provide AI with metaphysical/epistemic/metaethical axioms?
50:54 Epistemic deference to AI on concepts too complex for the human mind to grasp
53:29 Does AI understand stuff?
54:56 Human-like AI now b/c training data - perhaps alien-like AI soon based on new training regimes
57:30 Base Camp at Mt Ethics, cosmic norms as metaethics developed by a cosmic host
59:39 AI Creation & the Cosmic Host - train AI to defer to cosmic norms
1:03:20 Values as causal influencers of goals, so better get the values right
1:06:36 Deep Utopia if we get superintelligence right - what is Utopia
1:13:16 Shallow redundancy vs deep redundancy - subtle values coming into focus
1:14:12 Deep utopia may involve giving up some things. Workng through repulsiveness of utopain visions
1:14:39 Going back to basic constituents of values, from there rebuilding value
1:15:36 Philosophical particle accelerator
1:19:12 Taking the scenic route to posthuman forms instead of rushing
1:28:01 Hedonics and the 5 Ringed Defence of Meaning in a plastic utopia
1:40:20 The space of subtle values is enormous, Ikagai and Japanese culture
1:49:45 Maximum novelty experienced early in life?
1:52:04 Putting the picture together
1:55:34 Outriders, and Finding Purpose Now
1:57:04 Reflections on the old era of futurology - Conclusion
Elios AI eleosai.org
#ai #utopia #superintelligence #agi
The music at the end is 'Hedonium 1' which is guitar saturated with Strymon reverbs, delays and modulation: soundcloud.com/limping-fiddler/hedonium-1
Many thanks for tuning in!
Please support SciFuture by subscribing and sharing!
Buy me a coffee? buymeacoffee.com/tech101z
Have any ideas about people to interview? Want to be notified about future events? Any comments about the STF series?
Please fill out this form: docs.google.com/forms/d/1mr9PIfq2ZYlQsXRIn5BcLH2onbiSI7g79mOH_AFCdIk
Kind regards,
Adam Ford
- Science, Technology & the Future - #SciFuture - scifuture.org




![Wendell Wallach & Adam Ford - Utopia, Superintelligence & Cosmic Morality #AI #AIEthics #Ethics
Wendell Wallach and Adam Ford discuss their visions for the future, focusing on superintelligence, human evolution, and the possibility of a cosmic morality.
Key Discussion Points:
- Morally Motivated Superintelligence: The conversation begins with a reflection on what a utopia might look like. One perspective offered is a beneficial intelligence explosion where superintelligence is developed with moral motivations that accommodate human concerns rather than being alien or divorced from humanity [00:13].
- Evolutionary Enhancement: Wendell Wallach views superintelligence not as a separate computational system that replaces humans, but as an evolutionary enhancement scheme [00:44]. He advocates for upgrading human biological and cognitive capabilities through self-understanding rather than focusing on technological immortality for a select few [01:04].
- Fair Distribution of Capabilities: A central hope discussed is that if civilization gains the ability to do grand things, the resulting benefits and dividends should be spread equally and fairly across all of society [01:36].
- Cosmic Morality and the Long-Term Future: The speakers explore cosmic game theory, suggesting that if a moral arc is discoverable by any advanced civilization, humanity might have more in common with alien life than currently assumed [02:05]. This leads to a vision of a cosmic collective in the distant future where intelligent life thrives and connects across the universe [02:24].
Wendell Wallach (born 21 April 1946) is a world-renowned bioethicist and expert on the ethics and governance of emerging technologies, particularly artificial intelligence (AI) and neuroscience. Often referred to as a godfather of AI ethics, his work focuses on the challenges of aligning rapidly advancing systems with human values.
# Current Roles & Affiliations:
Wallach remains active in several prestigious academic and policy institutions:
- Yale University: Emeritus Scholar at the Interdisciplinary Center for Bioethics, where he chaired the Technology and Ethics study group for 11 years.
- Carnegie Council for Ethics in International Affairs: Carnegie-Uehiro Senior Fellow and co-director of the AI & Equality Initiative (AIEI), which addresses structural inequalities driven by AI.
- The Hastings Center: Senior Advisor for this nonpartisan research institute focused on the social and ethical issues of science and health.
- Other Fellows: Fellow at the Sandra Day O’Connor School of Law (Arizona State University) and the Institute for Ethics & Emerging Technology.Key Publications
# He has authored several foundational works that helped define the field of machine ethics:
- Moral Machines: Teaching Robots Right from Wrong (2008): Co-authored with Colin Allen, this is widely considered the first book to examine the challenge of building artificial moral agents.
- A Dangerous Master: How to Keep Technology from Slipping Beyond Our Control (2015): Explores the risks of unchecked technological development and proposes governance solutions. A revised edition with a new preface was released in 2024.
- Library of Essays on Ethics and Emerging Technologies: Series editor for this comprehensive eight-volume collection published by Routledge.
# Global Policy & Influence
Wallach has a significant presence in international policy forums:United Nations: Has provided testimony to the UN on lethal autonomous weapons systems and advised on digital cooperation.
World Economic Forum (WEF): Formerly co-chaired the Global Future Council on Technology, Values, and Policy and served on the AI Council.
# Awards: Recipient of the World Technology Network awards for Ethics (2014) and Journalism & Media (2015).
#moralmachines #MachineEthics #AI
Many thanks for tuning in!
Please support SciFuture by subscribing and sharing!
Buy me a coffee? https://buymeacoffee.com/tech101z
Have any ideas about people to interview? Want to be notified about future events? Any comments about the STF series?
Please fill out this form: https://docs.google.com/forms/d/1mr9PIfq2ZYlQsXRIn5BcLH2onbiSI7g79mOH_AFCdIk/
Kind regards,
Adam Ford
- Science, Technology & the Future - #SciFuture - http://scifuture.org Wendell Wallach & Adam Ford - Utopia, Superintelligence & Cosmic Morality #AI #AIEthics #Ethics](https://i.ytimg.com/vi/yrjH6lrbGU8/mqdefault.jpg)





