Uploaded April 2023 | Updated September 2026, 3 days ago
Abstract:
Large generative models create the potential for many beneficial use cases, but they also raise significant ethical and social risks of harm. How can we know whether a large model is aligned with our expectations? In this talk, we first present a taxonomy of ethical and social risks developed by a multidisciplinary group of DeepMind researchers. We then turn to evaluation approaches to measure these risks. Surveying existing evaluation approaches - such as automated benchmarking and human adversarial testing, reveals that current evaluation frameworks have significant gaps and limitations. We highlight what the field is not standardly evaluating (yet) and propose approaches toward closing these gaps. We close by highlighting directions for future research.
Bio:
Laura Weidinger is a Senior Research Scientist at DeepMind, working on Ethics research. Laura's academic background is in cognitive neuroscience and philosophy. At DeepMind, Laura’s research focuses on identifying the link between early-stage design decisions on AI systems and their downstream social implications. To this end, Laura builds analysis and evaluation frameworks for AI, and runs targeted studies with human participants. Currently, Laura is working on evaluating ethical risks from large language models, and on human interaction with large generative models.
scholar.google.com/citations?user=SFQLTCkAAAAJ&hl=en
Abstract:
Large generative models create the potential for many beneficial use cases, but they also raise significant ethical and social risks of harm. How can we know whether a large model is aligned with our expectations? In this talk, we first present a taxonomy of ethical and social risks developed by a multidisciplinary group of DeepMind researchers. We then turn to evaluation approaches to measure these risks. Surveying existing evaluation approaches - such as automated benchmarking and human adversarial testing, reveals that current evaluation frameworks have significant gaps and limitations. We highlight what the field is not standardly evaluating (yet) and propose approaches toward closing these gaps. We close by highlighting directions for future research.
Bio:
Laura Weidinger is a Senior Research Scientist at DeepMind, working on Ethics research. Laura's academic background is in cognitive neuroscience and philosophy. At DeepMind, Laura’s research focuses on identifying the link between early-stage design decisions on AI systems and their downstream social implications. To this end, Laura builds analysis and evaluation frameworks for AI, and runs targeted studies with human participants. Currently, Laura is working on evaluating ethical risks from large language models, and on human interaction with large generative models.
scholar.google.com/citations?user=SFQLTCkAAAAJ&hl=en
