Uploaded July 2026 | Updated November 2026, 1 day ago
Dr. Leonard Dung is a Postdoctoral Researcher at the Institute of Philosophy II, Ruhr-University Bochum
See paper: philarchive.org/rec/DUNISN
Abstract : Numerous authors have expressed concern that advanced artificial intelligence (AI) poses an existential risk to humanity. These authors argue that we might build AI which is vastly intellectually superior to humans (a ‘superintelligence’), and which optimizes for goals that strike us as morally bad, or even irrational. Thus this argument assumes that a superintelligence might have morally bad goals. However, according to some views, a superintelligence necessarily has morally adequate goals. This might be the case either because abilities for moral reasoning and intelligence mutually depend on each other, or because moral realism and moral internalism are true. I argue that the former argument misconstrues the view that intelligence and goals are independent, and that the latter argument misunderstands the implications of moral internalism. Moreover, the current state of AI research provides additional reasons to think that a superintelligence could have bad goals.
- What is the core argument of "Is Superintelligence Necessarily Moral?"
- Can moral knowledge motivate artificial agents?
- How does this argument relate to Bostrom’s orthogonality thesis?
- Does moral realism change the alignment problem?
- What do current AI systems suggest about the relationship between learning, goals, and motivation?
Dr. Leonard Dung is a Postdoctoral Researcher at the Institute of Philosophy II, Ruhr-University Bochum
See paper: philarchive.org/rec/DUNISN
Abstract : Numerous authors have expressed concern that advanced artificial intelligence (AI) poses an existential risk to humanity. These authors argue that we might build AI which is vastly intellectually superior to humans (a ‘superintelligence’), and which optimizes for goals that strike us as morally bad, or even irrational. Thus this argument assumes that a superintelligence might have morally bad goals. However, according to some views, a superintelligence necessarily has morally adequate goals. This might be the case either because abilities for moral reasoning and intelligence mutually depend on each other, or because moral realism and moral internalism are true. I argue that the former argument misconstrues the view that intelligence and goals are independent, and that the latter argument misunderstands the implications of moral internalism. Moreover, the current state of AI research provides additional reasons to think that a superintelligence could have bad goals.
- What is the core argument of "Is Superintelligence Necessarily Moral?"
- Can moral knowledge motivate artificial agents?
- How does this argument relate to Bostrom’s orthogonality thesis?
- Does moral realism change the alignment problem?
- What do current AI systems suggest about the relationship between learning, goals, and motivation?










