What an Ex-Anthropic Researcher's Warning About Human Extinction Really Means

A string of posts on X from an AI researcher who says he quit a job at Anthropic over ethical concerns has gone viral, prompting thousands of responses and reigniting debate over the potential future dangers of artificial intelligence.

AI Companies Think Their Creations Could Kill Us All

Jacob Coxon, an AI researcher who also worked at OpenAI on pretraining AI models, resigned from Anthropic over concerns that AI development could lead to human extinction. "Neither company is acting responsibly," he wrote. "They are racing to self-improving superintelligence and gambling with our lives." Superintelligence refers to a hypothetical scenario in which AI's "cognitive" abilities vastly exceed those of humans across every domain - science, math, strategy, creativity, and problem-solving.

The Statement Quickly Spread on Social Media

The threaded post on X from Coxon has more than 156 million views as of this writing and has been liked more than 752,000 times. The statement quickly spread on social media and was featured on major news sites. It even drew a comment from Illinois Gov. JB Pritzker, who responded on X, "It's becoming more clear the threat AI poses to humanity, so I'm calling for immediate action from the industry and Washington."

Others at Anthropic Confirmed the Threat

In less than 24 hours, the statement also drew responses from others at Anthropic who, rather than debating with Coxon, confirmed that the threat that AI will overtake and potentially eliminate humans is not a wild fantasy but a reality for AI researchers. "Jacob [Coxon] is correct here," wrote Evan Hubinger, a team leader at Anthropic in AI alignment stress testing, on X. Hubinger wrote in his response, "We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade."

Resignations That Act as Warning Signs

It's difficult to dismiss Coxon's concerns for one big reason: He's not the first researcher at a major AI company to leave for these reasons. In February, a safety lead at Anthropic, Mrinank Sharma, resigned from the company, telling followers on X, "The world is in peril … We appear to be approaching a threshold where our wisdom must grow in equal measure to our capacity to affect the world, lest we face the consequences."

Industry Leaders Are Rushing into the Unknown

That competition, with both OpenAI and Anthropic hurtling toward initial public offerings, has raised concerns that, instead of moving cautiously in deploying AI models, industry leaders are rushing into the unknown and releasing models that could pose present and future dangers. This was the case when Anthropic released advanced AI models, which ended up being pulled or temporarily restricted due to major cybersecurity risks.

AGI and Superhuman AI

The speed at which AI poses an imminent threat to humanity may depend on when the technology reaches two purported benchmarks that have become increasingly blurred: superhuman AI (also known as artificial superintelligence or ASI), which Coxon mentioned in his viral post, and AGI, or artificial general intelligence. AGI would mark the hypothetical point at which AI meets (or exceeds) the capabilities of a human mind.

Doomers vs. Boosters

The debate over whether or not AI will kill all humans by 2030 mirrors the broader discussion over AI doomers versus boosters, and the lack of nuance in the middle. While AI boosters typically hype AI by emphasizing its extraordinary potential, such as huge productivity gains, AI doomers hype it in the opposite direction, emphasizing job loss and human extinction.

The AI Industry's PR Crisis

The speed at which Coxon's post went viral and the response it's drawing may be an indication that the AI industry has another PR crisis on its hands in a year when AI data centers are under fire and trying to change their image, and social media companies, including Meta, are reeling from multibillion-dollar judgments over issues like tech addiction.