Uploaded February 2025 | Updated September 2026, 2 weeks ago
Title: Against Optimization
Abstract: Value is complex and hard to measure. So we often measure and optimize a simplified proxy metric. It’s easy to feel uneasy about this. For example, Nguyen (2024) describes this phenomenon as “value capture” and thinks it is bad. Similar phenomena are discussed as “Goodhart’s law” and “reward hacking”. But it is difficult to pinpoint exactly what is going wrong here. I will (i) discuss a theorem which attempts to give a general explanation of why optimizing a proxy metric is bad, (ii) ask whether the theorem rests on plausible assumptions and (iii) speculate about what this means for consequentialism, capitalism and various other topics.
Title: Against Optimization
Abstract: Value is complex and hard to measure. So we often measure and optimize a simplified proxy metric. It’s easy to feel uneasy about this. For example, Nguyen (2024) describes this phenomenon as “value capture” and thinks it is bad. Similar phenomena are discussed as “Goodhart’s law” and “reward hacking”. But it is difficult to pinpoint exactly what is going wrong here. I will (i) discuss a theorem which attempts to give a general explanation of why optimizing a proxy metric is bad, (ii) ask whether the theorem rests on plausible assumptions and (iii) speculate about what this means for consequentialism, capitalism and various other topics.
