The real reason AI researchers suddenly want to slow down

For years, AI safety organizations have warned that the big labs are racing to develop more intelligent and autonomous systems without a clear and realistic strategy for managing the risks. Independent researchers have argued that investments in safety and alignment lag far behind investments in capabilities. Employees have quit. Experts have predicted catastrophe. And none of it did much to slow the race. But over the past few weeks, the warnings have started to sound different-they’re increasingly coming from people working inside the labs. Last week, AI researcher Jacob Coxon announced his resignation from Anthropic in a post that got more than 171 million views on X. Shortly after, Anthropic Alignment Science Lead Evan Hubinger publicly agreed with Coxon, as did Anthropic alignment researcher Ethan Perez and scalable oversight researcher Samuel Marks. OpenAI safety researchers Julie Steele and Jasmine Wang also came out in support. So why now? At least part of the answer is written plainly in Coxon’s tweet: “They are racing straight to self-improving superintelligence and gambling with our lives,” he wrote. Coxon is referring to a technical concept called “recursive self-improvement,” or the use of existing AI models to build and optimize new AI models. Researchers can now use AI models in a number of parts of model development process. They can use AI models to design new computing infrastructure that delivers more computing power, efficiency, and processing speed. AI models can be used to create more and better training data, or manage and optimize the whole software framework that governs model training. Or, the AI might be used to write and optimize the code that defines and implements the model itself. In other words, AI models are not just getting better; they are beginning to take over the work involved in making the next generation of AI better. OpenAI, for instance, recently said its coding agents are already “meaningfully accelerating research progress” inside the company. By mid-August, its research organization was using 3.1 agent-workdays for every human workday, and the company said it had reached what it calls an “automated research intern.” Anthropic has similarly said that frontier AI models are now contributing to the development of their successors. Creating AI models good enough to take over these tasks is one reason that Anthropic, OpenAI, and Google have been so focused on developing AI coding assistants such as Claude Code, Codex, and Antigravity. Engineering departments within all kinds of enterprises have seized on these tools to accelerate their software development, and that’s provided a much-needed revenue stream for the AI labs. But inside the labs, the same systems can also be used to accelerate the development of new AI models.