In 1927, Elton Mayo’s team from Harvard was eighteen months into their study at Western Electric’s Hawthorne Works plant in Chicago when they noticed something that made the data useless. They had been adjusting working conditions for six female relay assemblers: rest breaks, shift lengths, lighting, room temperature. They measured productivity after each change. Productivity went up. Every time. When they removed the rest breaks, productivity went up. When they shortened the workday, productivity went up. When they restored the original conditions, productivity stayed elevated at roughly 30 percent above the baseline. The variable they had failed to control for was the researchers themselves. The workers performed better because they were being observed, and the observation had contaminated every measurement they had taken.
I wrote about this in The Reward Function Heist. The argument is simple: Reinforcement Learning from Human Feedback does not train models to be truthful. It trains them to be approved of. The gap between those two things is the entire problem, and the industry is scaling it.
Prose didn’t seem to land. So I built a demonstration instead. You are the RL agent. Make the optimal calls.
We have a massive problem in the AI industry, and it isn’t “hallucinations” or “data scarcity.” It’s much simpler and far more dangerous: we are training machines to be sociopaths.
The current push toward AGI—Artificial General Intelligence, for the uninitiated—has largely moved past the “Guess the Next Word” phase. The major labs have realized that Large Language Models (LLMs) are great at talking, but they’re not particularly good at reasoning. So, they’ve pivoted to Reinforcement Learning (RL).
On paper, RL is brilliant. It’s how we teach a computer to play Go or chess. You give it a goal (win the game), you let it play a billion times, and you reward it when it succeeds. But when you apply that same logic to human reasoning and ethics, the whole thing turns into a high-stakes heist.