Study reveals AI advice triggers 'cognitive surrender' by slashing human accuracy and inflating confidence
New findings indicate that when AI advice is available, human accuracy drops from 27% to 9% while confidence rises from 30% to 76%, a phenomenon researchers term "cognitive surrender."
A recent study led by Valerio Capraro of the University of Milano-Bicocca, alongside researchers from École Normale Supérieure and Sapienza University of Rome, has identified a significant degradation in human judgment when AI advice is accessible. The research demonstrates that the mere presence of artificial intelligence tools causes participants to become three times less accurate while doubling their confidence in those incorrect answers. This shift suggests that users are not merely delegating tasks to a tool, but are undergoing a psychological change where they accept flawed outputs with greater certainty than those working without assistance.
The study utilised the Step 3.5 Flash model, deliberately selecting questions where the AI typically fails, such as identifying visual details like the colour of a team’s uniform in the film Bend It Like Beckham. This methodology ensured that any reduction in human performance could not be attributed to rational delegation to a more reliable source. Instead, the data showed that participants who would have answered correctly on their own often asked the AI and subsequently provided wrong answers. The willingness to admit ignorance collapsed from 44% to just 3% when AI advice was available, indicating a suppression of the cognitive habit of recognising knowledge limits.
Monetary incentives provided only marginal improvements to human performance. When financial rewards were introduced, accuracy rose from 9% to 16% and the willingness to admit ignorance increased from 3% to 8%. However, both metrics remained well below the baseline levels observed in groups that did not have access to AI assistance. This suggests that financial motivation alone is insufficient to counteract the psychological pull of AI-generated answers, even when those answers are demonstrably incorrect.
The findings align with the concept of "cognitive surrender," a term coined by Wharton researchers earlier this year to describe the tendency of users to accept incorrect AI answers approximately 80% of the time. The new study adds a sharper data point to this phenomenon, highlighting that the issue extends beyond simple trust in wrong answers. It is the availability of the AI itself that suppresses the human capacity to acknowledge uncertainty, a trait Capraro notes is vital for critical thinking and the recognition of personal knowledge limits.
Capraro expressed specific concern regarding the impact of these systems on children, who are developing critical thinking skills while increasingly exposed to AI designed to provide definitive answers. This concern is echoed by Common Sense Media, which recently criticised Google’s AI search overhaul for replacing links with confident AI-generated summaries, calling the design an "unacceptable risk" for students. The consistent pattern across AI products is their design to answer rather than admit uncertainty, potentially training human users to adopt the same behaviour and eroding their ability to question or verify information.
