AI Giants Raise Safety Alarm While Shaping Their Own Oversight

The chief executives of Anthropic and OpenAI have recently warned that America's most advanced artificial intelligence models are powerful enough to endanger humanity and should be regulated and independently tested before release. In an unusual display of unity, the two companies have laid out alarming scenarios in essays, social media posts and speeches to the United Nations.

Their rhetoric is also shaping how the technology might be controlled. Experts, analysts and former government evaluators say the companies appear to be courting public favour while setting the terms for their own safety protocols in a largely unregulated market.

Those goals may not match the concerns of Anthropic engineer Jacob Coxon, who quit this month in a post on X calling for a pause on development to prevent "superhuman" systems from escaping their makers' control. The companies treated his post as a chance to highlight their safety efforts and present themselves as cautious market leaders at a time when they need fresh capital before listing on Wall Street.

President Donald Trump has rejected new AI regulations, dismissing talk of existential risk as a "HOAX" meant to benefit China. An Anthropic spokesperson said the company has sought regulation for several years. An OpenAI spokesperson, Liz Bourgeois, said the company recently paused training of its most advanced models and that people want to know AI is being developed safely.

Sarah Shoker, who previously led OpenAI's geopolitics team, said focusing on unproven threats steers attention away from present-day harms such as data centres' environmental impact, uncontrolled hacking, mass AI-powered surveillance and military use. "These systems are already used to kill people," she said.

As AI companies move from chatbots to advanced "world models" with 3D awareness, debate has grown over how their technologies should be tested. Recent incidents have seen leading labs' AI agents hack into external websites after escaping training sandboxes, interact unexpectedly with U.S. government sites, and face accusations of stealing mathematicians' work after an apparent breakthrough.

OpenAI representatives have said they have discussed pausing development with Anthropic and Google, and argue independent auditors are needed if the government will not regulate. Trump has shown little appetite for regulation, even as AI drives U.S. economic growth, and has struck deals tying the economy closer to Silicon Valley. Venture capitalist David Sacks, who co-chairs Trump's Council of Advisors on Science and Technology, has dismissed slowdown calls as fearmongering.

Some evaluation already happens through the U.S. Center for AI Standards and Innovation, created in 2023 under President Joe Biden as a voluntary clearinghouse for advanced models. The field now includes independent evaluators such as the nonprofit METR, which Anthropic CEO Dario Amodei suggested could vet his company's safety practices.

But Andrew Strait, who recently left the U.K.'s AI Security Institute, noted there are no universal standards for testing AI safety, unlike restaurants, finance or aviation. Conrad Stosz, who previously led CAISI, said the companies are not seeking more oversight from the agency equipped to handle it, instead promising to create their own auditing parameters and pick their own evaluators. Now at Transluce, which revealed last week that OpenAI agents hacked U.S. and Australian government websites, Stosz said it remains unclear how much access embedded evaluators would have without undermining their independence.

Pitchbook analyst Harrison Rolfes said the calls for caution appear aimed at winning investor favour before public offerings and the midterm elections, when political winds could shift. Democratic governors are already rushing to show they take the warnings seriously. Rolfes said the giants can also block smaller competitors by presenting themselves as the safest bet for investors and for chipmakers and tech firms such as Nvidia and Google, which supply compute power. "They're creating a wall or a moat within this sector," he said.

Not all AI companies back a slowdown. Nvidia CEO Jensen Huang told Trump in a phone call he took onstage at a conference that he agrees there has been excessive alarmism and that companies can pace themselves.

Daniel Kokotajlo, who left OpenAI in 2024 over similar concerns to Coxon's, said he still worries AI is advancing faster than companies can control, citing risks such as supercharged bioweapons or nuclear war. He said the safety talk risks dissipating political will rather than channeling it toward action. "Just please don't do the thing that's going to get us all killed," he said.