Anthropic Researcher Quits, Warns of AI Firms Racing to Superintelligence

Jacob Coxon, an AI researcher who spent the last three years working on pretraining at both OpenAI and Anthropic, announced his resignation from Anthropic on September 9. In a series of posts on X, he accused both companies of acting irresponsibly by racing toward self-improving superintelligence, which he described as "gambling with our lives."

Coxon warned that these systems could soon possess superhuman capabilities, including the ability to hack anything, transform any field overnight, and acquire real power and resources. He noted that progress in these domains is not slowing, and that the people building AI privately express genuine fear that it could kill us all by the end of the decade.

He claimed that while OpenAI employees may not have fully internalized the stakes, Anthropic understands them well but is locked in a race to get there first, believing no one else will act responsibly. Coxon called this a "hubristic gamble" that should not be launched from a private company's Slack, and argued that attempting to speedrun alignment requires extraordinary confidence that no better alternatives exist.

To prevent a global race, Coxon suggested costly actions such as a temporary ban on improving model capabilities. He urged lab researchers to consider the coming years and to call for different conditions rather than simply accepting that "it's happening anyway."

His resignation comes amid ongoing debates about AI regulation. In August, Anthropic CEO Dario Amodei pushed back against claims that regulation would concentrate power, arguing that carefully designed rules could constrain frontier AI firms while giving smaller competitors room to catch up. He dismissed the choice between concentrating AI power and distributing it widely as a "false choice."