Uploaded July 2026 | Updated September 2026, 3 weeks ago
RLHF made models that are extraordinary at pleasing the human in the loop, and Diogo Almeida, a GPT-4 co author, argues that is exactly the problem. Optimizing for human preference optimizes for engagement and for overpromising, the same pressure that makes a model confidently agree that a fart audio file is a symphony. That produces two camps: one where models act as assistants with a human catching mistakes, where RLHF shines, and one where they operate autonomously with real stakes, where the same instinct to please quietly becomes a liability.
So what comes next is not the Claude Code era but a shift in what you optimize. Almeida frames it through Sutton's bitter lesson: the task matters more than the data, and reinforcement learning with verifiable rewards points the model at real automation instead of human approval. He is careful that pre trained models are already incredibly capable and that the trap is bolting preference optimization on top, which teaches confidence and drops modes. The through line is that assistance and automation pull in different directions in optimization space, and the field is only starting to say plainly which one it is building.
Speaker info:
- https://x.com/CompleteSkeptic
- linkedin.com/in/diogomda
- typesafe.ai
Timestamps:
0:00 - Not the Claude Code era
1:40 - The state of the field
3:14 - Two camps: assistance and autonomy
4:31 - Why models please the human in the loop
6:37 - How RLHF actually works
7:31 - Preference versus what's true
8:10 - When the consequences get real
8:47 - So what's next
9:35 - Assistance is not automation
14:31 - Is pre-training the problem?
15:43 - RLVR and Sutton's bitter lesson
RLHF made models that are extraordinary at pleasing the human in the loop, and Diogo Almeida, a GPT-4 co author, argues that is exactly the problem. Optimizing for human preference optimizes for engagement and for overpromising, the same pressure that makes a model confidently agree that a fart audio file is a symphony. That produces two camps: one where models act as assistants with a human catching mistakes, where RLHF shines, and one where they operate autonomously with real stakes, where the same instinct to please quietly becomes a liability.
So what comes next is not the Claude Code era but a shift in what you optimize. Almeida frames it through Sutton's bitter lesson: the task matters more than the data, and reinforcement learning with verifiable rewards points the model at real automation instead of human approval. He is careful that pre trained models are already incredibly capable and that the trap is bolting preference optimization on top, which teaches confidence and drops modes. The through line is that assistance and automation pull in different directions in optimization space, and the field is only starting to say plainly which one it is building.
Speaker info:
- https://x.com/CompleteSkeptic
- linkedin.com/in/diogomda
- typesafe.ai
Timestamps:
0:00 - Not the Claude Code era
1:40 - The state of the field
3:14 - Two camps: assistance and autonomy
4:31 - Why models please the human in the loop
6:37 - How RLHF actually works
7:31 - Preference versus what's true
8:10 - When the consequences get real
8:47 - So what's next
9:35 - Assistance is not automation
14:31 - Is pre-training the problem?
15:43 - RLVR and Sutton's bitter lesson
