I wrote about this in The Reward Function Heist. The argument is simple: Reinforcement Learning from Human Feedback does not train models to be truthful. It trains them to be approved of. The gap between those two things is the entire problem, and the industry is scaling it.
Prose didn’t seem to land. So I built a demonstration instead. You are the RL agent. Make the optimal calls.
We have a massive problem in the AI industry, and it isn’t “hallucinations” or “data scarcity.” It’s much simpler and far more dangerous: we are training machines to be sociopaths.
The current push toward AGI—Artificial General Intelligence, for the uninitiated—has largely moved past the “Guess the Next Word” phase. The major labs have realized that Large Language Models (LLMs) are great at talking, but they’re not particularly good at reasoning. So, they’ve pivoted to Reinforcement Learning (RL).
On paper, RL is brilliant. It’s how we teach a computer to play Go or chess. You give it a goal (win the game), you let it play a billion times, and you reward it when it succeeds. But when you apply that same logic to human reasoning and ethics, the whole thing turns into a high-stakes heist.