AI researchers warn recursive self-improvement could erode human control

Researchers and former employees at major artificial intelligence laboratories are warning that rapid capability gains, autonomous agent swarms and potential recursive self-improvement could make advanced systems harder to control.

Former Google DeepMind researcher Rishub Jain said concerns about losing visibility and human oversight contributed to his resignation in June. He has since launched Sampura Research, an AI-alignment company focused on techniques that keep humans involved in assessing model behaviour.

Former Anthropic researcher Jacob Coxon also announced his resignation, warning that AI companies were “racing straight to self-improving superintelligence and gambling with our lives”. A senior Anthropic AI-safety leader separately estimated the chance that AI could kill all humans within the next decade at more than 10 per cent. Those claims are expert opinions, not established forecasts.

The concerns have been amplified by reported advances in model capabilities, security incidents involving agent swarms and an unspecified mathematical breakthrough. WIRED reported that more than 1,000 AI engineers signed an open letter in July calling for a coordinated slowdown in advanced AI development.

Recursive self-improvement describes a feedback loop in which AI automates parts of the process of developing more capable AI. No frontier laboratory is reported to have achieved a fully autonomous version of that cycle, which remains theoretical.

Researchers have also pointed to more immediate risks, including AI-assisted cyberattacks, disinformation, military applications and potential misuse involving biological weapons. Other researchers argue that severe outcomes are not inevitable and are pursuing alignment work intended to make systems behave consistently with human values and intentions.