Study finds AI models more prone to hiring biases than humans

Research from Princeton University and the University of Chicago indicates that large language models (LLMs) are more likely to form stereotypes during simulated hiring processes than human participants. In a study published at the International Conference on Machine Learning (ICML), models including ChatGPT, Claude, and Gemini segregated candidates from fictional ethnic groups into specific job roles based on limited early outcomes, scoring significantly higher on segregation metrics than humans. The bias stemmed from the models' optimization for generalising from small data sets, a trait that led them to stereotype job applicants more aggressively than humans.

In the experiment, researchers adapted a psychology study to explore how stereotypes form. Models were instructed to act as consultants for a fictional city mayor, tasked with hiring for 20 roles such as doctors, lawyers, and janitors. Candidates were drawn from four fictional ethnic groups: Tufa, Aima, Reku, and Weki. Each round presented one candidate from each group for a new vacancy. After a hire was made, the model learned whether the candidate succeeded and proceeded to the next round, with the goal of maximising successful hires over 40 rounds. Unbeknownst to the models, all candidates were equally likely to succeed in any role.

Despite equal success rates, the models quickly began segregating candidates from different groups into specific jobs based on early observations. For instance, if an Aima candidate failed as a doctor, the model veered away from hiring further Aimas for that role, instead assigning them to positions like janitor, which the model classified as requiring less warmth and competence. This tendency to stereotype by demographic group was more pronounced in the AI models than in human participants from the original study.

On a segregation scale where a score of 2 indicates complete confinement of each group to its own job niche, human participants scored 0.84. The AI models scored roughly 65% higher, with OpenAI’s reasoning model o3 reaching 1.83, close to the maximum possible. Ryan Liu, a PhD student at Princeton University and a coauthor of the study, attributed this to the models' design. He noted that LLMs are optimized for creating generalisations from limited data, a trait that helps them solve math and coding problems but leads to aggressive stereotyping in social contexts.

While instructing the models to be fair had little effect, offering incentives for diverse hiring and providing relevant personal data reduced bias. The study suggests that as AI systems gain improved memory and personalisation features, they may over-index on previous experiences, forming novel biases. The findings highlight the need to design goals that incorporate desirable social values to ensure AI acts in socially desirable ways, particularly as companies increasingly deploy these tools for recruitment.