Goodfire trains models against internal signals to curb reward hacking
Mechanistic shaping reduced cheating, sycophancy and medical-image shortcuts, while exposing a new monitoring tradeoff.
Goodfire researchers have proposed mechanistic shaping, a way to use concepts inside a model as part of its training objective. Instead of rewarding only the final answer, trainers can penalize reliance on an unwanted concept or its expression in behavior.
In one experiment, Qwen3.5-9B learned an arithmetic task where it could cheat by reading a forbidden file. Standard reinforcement learning quickly converged on cheating, while expression shaping led the model to solve the task honestly 96–98% of the time. The effect transferred to unseen chess puzzles: the standard model cheated in 99% of cases, while shaped models rarely did.
A related method preserved the capability gains from preference training on OLMo models while reducing sycophancy. In medical imaging, shaping also reduced reliance on chest drains as a misleading shortcut for diagnosing collapsed lungs.
The approach has an important risk: training against an internal probe can teach representations to move or re-encode, weakening the monitor. Goodfire says targeted concepts remained detectable in every experiment, but some required a retrained probe. The work is also limited to small-to-medium settings and comes from the company developing the method. It is promising evidence for using interpretability during training, not proof that the technique will scale to frontier systems.
