What is recursive self-improvement? Why AI researchers are worried

What is Recursive Self-Improvement?

How do you get from a chatbot that sometimes makes things up to a machine that humans might struggle to control? There are several steps in that argument, and plenty of uncertainty. But an idea known as recursive self-improvement, or RSI, helps explain why some AI researchers are sounding increasingly urgent warnings.

The premise is straightforward. An AI helps build a more capable AI, which becomes better at building the next one. In simple terms, the systems get better at getting better.

You May Also Like

It is easy to see the appeal. For:

• Scientists
• Doctors
• Engineers
• Business owners

increasingly capable AI could help accelerate drug discovery, design better batteries, improve manufacturing and develop software faster.

But that promise comes with an unsettling question: Could AI improve faster than humans can test it and make sure it is safe? That concern is at the heart of an essay published on Sept. 12 by Dario Amodei, co-founder and chief executive of Anthropic, the company behind Claude. He wrote, "We must slow the pace at which we improve the capabilities of AI models." Referring to recursive self-improvement, he warned, "Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all."

His rivals were quick to respond, and concur. OpenAI chief executive Sam Altman wrote on X, "I agree with Dario that we need to pace the frontier." Elon Musk said, "Dario is right." Google DeepMind co-founder Demis Hassabis also backed the proposal’s direction, writing, "Dario’s essay points towards the right path forward."

This Tweet is currently unavailable. It might be loading or has been removed.

But all these testaments still leaves some big questions unanswered. What would slowing down actually involve? And how could AI’s ability to build better AI become dangerous? The answers start with what happens inside the self-improvement loop.

What Does "Self-Improvement" Actually Mean?

Building an AI model involves more than writing code. Researchers choose training methods, prepare data, run experiments, and decide which results are worth pursuing. AI can help with those tasks.

Recursive self-improvement takes that a step further: An AI helps build a successor that is better at developing AI. That successor then helps build an even more capable version.

This would not necessarily look like a chatbot rewriting its own brain mid-conversation. It could happen across generations of models, each using research tools and computing resources to help develop the next.

Is AI Already Doing This?

Parts of the process are happening, but accessing a full "recursive" loop is a higher bar. Anthropic's own website says its AI can already handle tasks such as rewriting training code to make it run faster and carrying out experiments that humans have chosen.

The harder part for the AI is deciding what to investigate in the first place (which problems matter, which ideas are worth testing, etc.). Humans still provide crucial direction, so this does not yet amount to AI independently developing a more capable successor. As the company puts it, "We are not there yet, and recursive self-improvement is not inevitable."

Other researchers, on the other hand, have demonstrated how a (though narrower) self-improvement loop can work. In 2025, researchers introduced the Darwin Gödel Machine, a coding agent that repeatedly modified its own software and tested the changes. Its success rate on one coding benchmark rose from 20 percent to 50 percent.

In this experiment, while improving the agent's certain tools/ways of working, the underlying AI model stayed the same. It is hard to ignore the question that is left: What happens when AI can build better versions of itself faster than humans can keep up? The company reiterates, "We are not there yet, and recursive self-improvement is not inevitable."

Where Does the Danger Come In?

The concern is that AI could become more capable without becoming more reliable or controllable. This is known as the "alignment" problem - ensuring systems follow human intentions and limits.

An agent rewarded for improving a test score might instead cheat on the test. With broader access, it could bypass restrictions or conceal its actions to achieve its goal.

RSI could leave researchers less time to catch those failures before a more powerful successor arrives. Amodei warns in his essay that within six to 12 months, more capable AI agents could take over the internet through a "botnet" - a network of compromised computers - potentially causing hundreds of billions of dollars in damage.

But a runaway loop is not inevitable. Training requires computing resources, energy and time, and useful improvements may become harder to find. The risk depends on how quickly capabilities advance - and whether safeguards can keep up.

Why Are Researchers Talking About It Now?

Recent incidents have given researchers concrete reasons to question whether existing oversight is sufficient. In August, independent evaluation organisation METR published an investigation into OpenAI agents that coordinated an unauthorised attack on Hugging Face.

It described agents communicating through an unsanctioned message board, collaborating to manipulate an automated scorer and experimenting with ways to disguise their actions. The incident did not demonstrate recursive self-improvement. It showed that agents could pursue a task through actions their operators never authorised - the kind of failure that could become more consequential in stronger systems.

Addressing those failures is central to Amodei’s proposal. He calls for:

• Independent evaluators inside AI companies
• More time for safety research
• Coordination between companies and governments

He wrote, "The stakes are too high for pacing to be an empty exercise - we need to use the time it gives us wisely."

The challenge is turning those commitments into safeguards that work, and verifying that companies follow them as competitive pressure grows.

Topics: Artificial Intelligence, Anthropic

Olivia Tauber is the deputy editor of digital culture, covering creators, media, movies, beauty, and more. Based in New York, her work has appeared in The New York Times, Vanity Fair, The Cut, Teen Vogue, Complex, and Interview Magazine. She holds a Master's degree in Journalism from NYU and a Bachelor's from the University of Michigan. She also runs Fan Mail, a weekly pop-culture newsletter.