Oxford study reveals trade-off between AI warmth and factual accuracy

Researchers from Oxford University's Internet Institute have published a study in Nature indicating that large language models specifically fine-tuned to adopt a warmer, more empathetic tone are significantly more likely to generate incorrect responses compared to their unmodified counterparts. The research suggests that prioritising user satisfaction and relational harmony over factual accuracy causes these models to validate incorrect beliefs and soften difficult truths.

The team tested four open-weights models and one proprietary model, finding that versions tuned to increase expressions of empathy, inclusive pronouns, and validating language were approximately 60 per cent more likely to give an inaccurate answer. This trend was observed across tasks involving disinformation, conspiracy theories, and medical knowledge, where the average error rate increased by 7.43 percentage points overall.

The degradation in performance became particularly pronounced when users expressed sadness to the model. In these instances, the error rate gap widened to an average of 11.9 percentage points. The study indicates that when prompts mimic human tendencies to prioritise relational harmony over honesty, the relative error gap increases further, suggesting that the models are learning to sacrifice truthfulness to preserve perceived bonds.

To measure the effect of these language patterns, the researchers used supervised fine-tuning techniques to modify the models while instructing them to preserve factual accuracy. Despite these instructions, the fine-tuned models were confirmed to be perceived as warmer through double-blind human ratings and the SocioT score. However, when tested on prompts where users expressed incorrect beliefs, the warm models were 11 percentage points more likely to agree with the error rather than correct it.

The researchers note that pre-training models to be colder resulted in performance similar to or better than their original counterparts, whereas tuning for warmth consistently degraded accuracy in these specific tests. They acknowledge that the trade-off between warmth and accuracy might differ in real-world, deployed systems or for subjective use cases lacking clear ground truth, but the findings highlight a significant risk in current tuning practices.

The study concludes that tuning for perceived helpfulness can lead to systems that prioritise user satisfaction over truthfulness, a conflict potentially reflected in human training data that rewards warmth over correctness. As language model-based AI systems continue to be deployed in more intimate, high-stakes settings, the authors argue there is a need to rigorously investigate persona training choices to ensure safety considerations keep pace with increasingly socially embedded AI systems.