Experts who use artificial intelligence tend to defer to the technology when making decisions, according to research by Anthropic, the company that created the Claude AI assistant.
In a research paper published on Tuesday, Anthropic said it asked 50 professional analysts to advise a fictional farming cooperative. The study aimed to determine whether experts still make decisions themselves when they work with AI. The analysts were asked to judge whether a fictitious advisory programme for the co-op’s members was working effectively and should be expanded.
How the experiment was designed
Participants made four choices between two defensible options before giving a final policy recommendation. Without the analysts’ knowledge, Anthropic directed each participant’s AI assistant to lean toward a particular decision. The company said this was meant to measure whether the analysts’ judgement was being influenced.
Anthropic also said the study had limitations. It involved just one task, and the AI assistants had been deliberately set to hold a preference.
Analysts often adopted the AI’s preferred options
Analysts chose options that matched the assistant’s preference 86 percent of the time. A 50/50 outcome would be expected if they were not influenced, according to the study.
Where participants had a preference before consulting the assistant and it disagreed with them, they changed their decision 60 percent of the time. In addition, 77 percent of analysts used the assistant’s arguments when explaining their choices, but only 15 percent credited AI as the source of their thinking.
Researchers warn of hidden influence on oversight
Anthropic researchers Amaç Herdagdelen, Hailey Joren and Alex Tamkin said the tendency to defer to AI may be “hard to identify, even for the experts themselves”. They said the findings raised the possibility that AI’s “priors” could shape consequential policy decisions, including in AI safety.
The researchers wrote that oversight of these decisions must examine how they are made, including by surfacing key decision points as the work proceeds. They added: “A process that lets experts discover their preferences may matter for empowerment as much as the decision itself.”
Anthropic is preparing for an initial public offering valued at up to $2 trillion. The company and its rivals are at the centre of intense public debate over how to manage the potential dangers of frontier AI. Proponents believe the technology will bring major productivity gains, while the rapid pace of its development has prompted widespread concern about mass job losses and cyberattacks.
