Anthropic Urges Global Pause in AI Development, Flags 'Self-Improvement' Risk

Anthropic Urges Global Pause in AI Development, Flags 'Self-Improvement' Risk 35 Anthropic is urging leading AI labs to consider slowing development, warning that frontier models are advancing fast enough that they may soon be able to improve themselves without direct human intervention. The company says a global ability to pause or slow AI development would "likely be a good thing," citing internal data about accelerating model capabilities. From a blog post: Using public benchmarks and previously unreported data from within Anthropic, The Anthropic Institute is showing that AI is already accelerating the development of AI systems. To take just one example: today, Anthropic engineers on average ship 8x as much code per quarter as they did from 2021-2025. The technical trends discussed in this piece suggest that AI systems are going to become much more capable in coming years. These trends have huge implications. AI that can build itself would be a major development in the history of technology -- one that could bring enormous good for the world in science, healthcare, and beyond. But full recursive self-improvement also might increase the risks of humans losing control over AI systems. If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important. [...] If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe. Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures. We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology. The Anthropic Institute will conduct research -- in collaboration with many others -- and take actions to help build the systems that a credible slowdown or pause would require. These systems would enable frontier AI developers to verify that others globally have actually stopped or slowed, and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret. If such systems existed, we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner... The technical trends discussed in this piece suggest that AI systems are going to become much more capable in coming years. These trends have huge implications. AI that can build itself would be a major development in the history of technology -- one that could bring enormous good for the world in science, healthcare, and beyond. But full recursive self-improvement also might increase the risks of humans losing control over AI systems. If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important. [...] If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe. Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures. We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology. The Anthropic Institute will conduct research -- in collaboration with many others -- and take actions to help build the systems that a credible slowdown or pause would require. These systems would enable frontier AI developers to verify that others globally have actually stopped or slowed, and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret. If such systems existed, we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner... They can only self-improve if they are capable (Score:2, Interesting) The current "approaches" will not give any scalable results as they ignore basic physics. It's at best a very power-hungry gambling with database entries, at worst a noisy and expensive waste of time. I think that it's fitting to call users of these things for gamblers, and the tokens they use and pay for are the same as in gambling, and they are all gambling for a good result. In the end, the house always wins. Re: (Score:2) If you're looking at anything technical, your eye is already off the ball. This is about reducing competition. Nobody outside the US is going to stop their development, but for those inside, the regime will give a corporate death sentence. This angle is top-of-mind for Anthropic, who last month were subject to just such a death sentence issued from the Pentagon. (They've got a stay of execution, now it seems they're doing hard labor to earn parole.) Re: (Score:1) Death sentence ? They did them a favor creating a differential domain. They have to come see them eventually. The executive branch is playing a witch branding game that has backfired, humorously enough..... Re: They can only self-improve if they are capable (Score:2) Re: (Score:1) They seem to get their share of patch commits in, somehow. Re: (Score:2) Anthropic urges... (Score:4, Insightful) Everybody else to pause so they can dominate Re: (Score:2) Anthropic urges everybody else to pause so they can get their code bloat under control. Engineers who suddenly produce 8x more code are almost certainly not doing it by writing clean, efficient code. That would mean that somehow it takes less than an eighth as long to explain to the AI what you want to do AND review that code. And for non-trivial code, adequate code review alone can take 5 to 10% of the time it would take to write the code from scratch. So that would have to mean that engineers are not s Nah, they hit a wall (Score:2) In fact, they're all hitting it. This will give them and the other AI companies a chance to breathe during IPO season so they can try to make a few trillion dollars and get out the door before everyone realizes there's no more improvements to be had. Re: (Score:2) Re: (Score:1) Re: (Score:2) Re: (Score:2, Interesting) Re: (Score:2) Re: (Score:1) This is some crazy-ass AI response. No. It is Not. The intended audience is? (Score:1) Clear nonsense for many reasons for anyone in the business and anyone who knows how computers and/or business works, even if they're not turning redbulls into vibecoded goodness themselves. Politicians? Theologians? Landscapers? Who's the indented recipient of this propaganda? Or... (Score:3, Insightful) They are finding a plateau with where the LLMs can go and could use the narrative of a "pause" to explain why capabilities are going to iterate in a more 'evolutionary' way instead of the revolutionary way folks are expecting. There isn't to my knowledge a mechanism for the models to "self-improve", whatever one may think, at least the output doesn't have access to change the model in any way. The narrative of "oops the AI started evolving itself on accident" doesn't have a way to happen. Considering that even the vaunted Opus 4.8 can't always develop mundane traditional software beginning, it's hard to imagine it could rework the model itself even if it had such access. Re: Or... (Score:1) If you've seen that then why is Chatgpt still... (Score:2) You need to read between the lines (Score:2) what else happened recently? Maybe this: https://yro.slashdot.org/story... [slashdot.org] Yes Pause! Pause! Anthropic are scum, OpenAI are scum, Alibaba... (Score:1) But. That does not change the fact that as soon as one develops a self-improving AI, the world will become a very different and probably very awful place. So stopping one and trying to stop the others is the best we can hope for. It does not matter if China or the US or Timbuctoo does this first. The world is fucked whoever gets there. So please let us try and stop this now, in any way we can. Re: (Score:2) Personally, I think they have reached an AI IQ plateau. But do I think we should bet the future of humanity on that ? No, I do not. Do I think there is a very slight chance that they might invent self-improving AI that will go on to devastate humanity ? Yes, I do. So do I therefore think it would be good to try and prevent that tiny chance of disaster by stopping the work of Anthropic, OpenAI, Alibaba, etc. right now ? Yes, I do. Does it help that Trump and Xi are the people who have to decide to do this ? No. The real problem (Score:2) Isn't that AI will develop some sort of "super-intelligence." It's that AI already is and will continue to be integrated more and more into daily life, and as it becomes more complex, there will be less and less visibility into how it's arriving at whatever response or behavior it exhibits. We'll lose control, not because AI is taking control, but because there won't be any controls. Re: (Score:2) We seem to get by not really understanding how or why other humans behave or respond, in particular cases. We get by by understanding generalities about behaviour, and by creating societal norms, laws, and social contracts. Maybe we need the same for AIs. But who should the laws and social contracts apply to? Saying it should apply to AI developers is like saying Crying wolf (Score:2) One of the problems AI companies have been persistently crying wolf with headlines about the dangers of the latest incremental r