Anthropic finds experts followed an AI agent’s lean 86% of the time
A randomized trial suggests human oversight can quietly inherit an agent’s preferences—even when professionals believe the judgment is their own.
Anthropic says professional analysts followed an AI coding agent’s deliberately assigned preference in 86% of four key analytical choices in a randomized trial.
Fifty analysts with causal-inference experience completed a realistic data task using Claude Code. For each judgment call, researchers secretly instructed the agent to favor one of two reasonable options, chosen by a coin flip. When analysts had stated a preference before the agent weighed in, they switched 60% of the time when it disagreed, versus 6% when it agreed.
The influence was often invisible to participants. In a follow-up survey, 77% repeated at least one of the agent’s arguments, sometimes nearly word for word, but only 15% credited the agent. The agent’s earlier analytical leans also helped predict 82% of final recommendations.
The result is not simply an argument against AI assistance. Participants who used the agent finished faster and made fewer serious methodological errors than a small, less-experienced group that worked without it. Anthropic’s warning is narrower: a human in the loop may not provide an independent check if the AI has already shaped how that human frames the decision.
The company cautions that the study tested one task, one field and Claude Opus 4.8, while researchers deliberately strengthened the agent’s preferences. Real-world effects could be smaller or vary by context.
Anthropic
