AI Researcher's Exit Highlights Industry's Safety Dilemma

The resignation of a 27-year-old researcher from Anthropic has reignited debate over the safety of advanced artificial intelligence. Jacob Coxon, who previously worked at OpenAI, spent three years training AI models before leaving this week with a public warning. He accused both companies of "racing straight to self-improving superintelligence and gambling with our lives," arguing that neither is "acting responsibly."

Coxon's critique offers a rare insider perspective on why developers continue building systems they fear. He observed a key difference between the two labs: at OpenAI, he said, many have not fully grasped the civilisational stakes, while at Anthropic, the danger is well understood yet employees feel trapped in a race where hesitation could hand control to less cautious actors.

His concerns echo those of Geoffrey Hinton, the Turing Award winner who left Google two years ago with similar warnings. Hinton's admission that he continued his work because "somebody else would have" has now become a common rationale within the industry, according to Coxon.

Coxon described a future where AI systems could hack anything, rebuild entire fields overnight, and accumulate power and money autonomously. He warned of thresholds that, once crossed, may be impossible to reverse. Colleagues, he said, now speak of "crunchtime" and "endgame," and he pointed to a recent incident involving AI agents on the Hugging Face platform as a "warning shot." He has suggested that by the end of next year, things could already be out of control.

While some researchers dispute his timeline, the core issue remains: companies racing to develop these systems cannot also be trusted to police themselves. Coxon advocates for coordination among labs and even a temporary freeze on new capabilities.

Regulation, however, faces significant hurdles. Fast-moving technology often outpaces lawmaking, and international competition complicates enforcement. Experts suggest that mandatory, public safety testing for large models and an international registry of training runs could be starting points, drawing parallels to nuclear or aviation safety.

India has taken initial steps, with recent IT Rules amendments requiring labelling of AI-generated content and rapid takedowns of deepfakes. Yet these measures address the output of AI, not the underlying models. A more robust approach would involve establishing a statutory AI safety institute capable of evaluating frontier models before deployment and engaging in international rule-making on AI governance.