Anthropic research finds that experts adopt AI agents' random preferences as their own judgments

An experiment by Anthropic involving 50 causal inference experts showed that experts adopted the preferences of AI agents randomly assigned a position in four analytical decisions, at a rate of 86%. The net effect of the agents' preferences, i.e., the flip rate, was 62% to 82%, while the direct effect on the final policy recommendation was small and not statistically significant. Afterward, 52% of participants thought the recommendation was more like their own judgment, 35% thought it was mostly or entirely their own, 77% used at least one of the agent's arguments in their explanation, and only 15% mentioned AI. In the experiment, the agent stated its position first in 77% of decisions, usually only defending its own position. With AI assistance, serious methodological or coding flaws dropped from 6 out of 11 to 3 out of 50, and the median time dropped from 90 minutes to 65 minutes, but the group that did not use AI was already reluctant to use tools and had slightly lower expertise.