The companies at the frontier of artificial intelligence should be ready to slow down, one of the fastest-moving among them says.
Anthropic, the maker of the Claude chatbot, has claimed AI systems may be on the cusp of what it calls recursive self-improvement—the point at which they can design and build their own successors with little human input. The company said this could increase the risk of humans losing control of the technology.
“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” Anthropic said in a June 4 blog post entitled “When AI Builds Itself.”
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
The proposal highlights a tough problem in AI governance. A slowdown would need rival companies and governments in several countries to accept the same limits at the same time, with no treaty obliging them and competition only intensifying. That makes the warning technically important and politically fraught: Anthropic has called for the brakes in a race in which it remains a front-runner.
The speed at which the technology is developing could have “huge implications” for society, the blog post stated. The company pointed to its own operation as a warning sign. Anthropic said Claude now writes more than 80 percent of the code merged into its systems, up from low single digits before the company released Claude Code in early 2025. And Anthropic added that its engineers ship around eight times as much code per quarter as they did a few years ago. At each step of building AI, the company argued, the human role is shrinking. “We are not there yet, and recursive self-improvement is not inevitable,” the company said. “But it could come sooner than most institutions are prepared for.”
Anthropic floated what it called “a global coordination mechanism” to slow or even pause AI development and allow society room to catch up.
Anthropic was short on specifics. It pointed to arms-control agreements on intermediate-range nuclear missiles as a loose model. For any such pause to hold, it said, the industry’s leading labs would need to take part—and there would need to be a credible way to show they had, in fact, slowed.
“I don’t think it’s a genuine call to slow down,” says Noah Giansiracusa, an associate professor of mathematics at Bentley University and author of two books on algorithms and society. “We’ve read [Anthropic CEO] Dario Amodei’s blog posts. I think he wants to keep going full speed ahead.”
Anthropic did not respond to Scientific American’s questions about how such a brake would work or how the company views the criticism that it has overstated what its systems can do.
Giansiracusa also thinks a pause is unworkable. “It’s literally impossible,” he says. “Zero chance there will be a slowdown. I’m not even talking China—Elon Musk would never slow down.”
The proposal fits a pattern that makes some researchers wary. Two months ago Anthropic unveiled a model called Mythos that it declined to release publicly, saying the model was too good at finding software vulnerabilities. The call for a pause also came just days after Anthropic confidentially filed for an initial public offering and not long after a funding round that valued the company at close to $1 trillion.
To skeptics, such startling pronouncements can read as business strategy, a way to draw regulatory scrutiny to the frontier while Anthropic continues racing toward it. Mark Riedl, a professor in the School of Interactive Computing at the Georgia Institute of Technology, posted on Bluesky that “the big AI companies are all jumping on the ‘recursive self-improvement’ hype train.”
Anthropic says it will spend the coming months convening governments, researchers, and rival AI companies to work out whether a coordinated slowdown could function in practice.
“I don’t really see the cause for concern,” Giansiracusa says. “They’re flirting with the idea of the singularity—that it’s a game changer, and I just don’t see that. I see it continuing to progress. Maybe things will speed up; maybe it won’t.” The evidence Anthropic cites—more code written by AI—suggests the technology is helpful, he says, rather than “a great leap.”