“GAMBLING WITH OUR LIVES”

Anthropic Researcher Resigns, Warning that AI Race to Superintelligence Could Endanger Humanity

A researcher with the artificial intelligence research company Anthropic is quitting the AI industry entirely, citing fears that leading labs are racing to build systems they cannot control. Leading AI labs are racing toward superintelligence which could outpace AI safety, and they are gambling with humanity’s future, he warned.

Jacob Coxon, 27, announced his resignation through a series of posts on X. In a viral post, Coxon cautioned that the risks of advanced AI could become catastrophic within the next decade.

He wrote that the AI industry is currently not acting responsibly. Instead, he said they are racing straight toward self-improving superintelligence. He said the industry is “gambling with our lives”.

Coxon’s reportedly told media that aggressive scenarios could arrive soon, specifically, he warned things could be “out of control” by late next year. His central fear centres on recursive self-improvement directly.

“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” he said.

To those unfamiliar, this concept describes AI improving its own capabilities repeatedly. Essentially, the AI takes over enough research to accelerate its successors. As a result, development could outpace researchers’ ability to evaluate it. Coxon warned future systems could hack anything and acquire real power.

Coxon said he resigned over concerns about the technology’s potential to surpass human control. In a widely shared X post, he said the AI industry was more focused on competition rather than on implementing safeguards. He came to this realisation after spending the last 3 years doing research – training new AI models on vast amounts of data – at OpenAI and Anthropic.

Both Anthropic and OpenAI have been rapidly developing and releasing new AI models as they work toward creating artificial general intelligence (AGI). Long considered the holy grail of AI technology, an AGI model would theoretically have the intellectual capabilities of a human, including reasoning, common sense, and creativity. An artificial superintelligence would surpass humans. (Mashable SE Asia)

Coxon believes that the issue has grown beyond the scope of any single company. He suggests that we might need some serious collaboration across the industry or even with the government before self-improving AI systems become too difficult to manage.

“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon said. “No other human activity poses this level of danger,” he added, referencing the swift advancement of AI technology.

The Wall Street Journal reported that “Coxon said he is leaving the company because he doesn’t want to be part of an industrywide race to build AI systems capable of improving themselves, fearing they could eventually spiral out of control and threaten humanity.”

Source: TechJuice; Al Jazeera; DNA India

AI “Could Kill Us All by the End of the Decade”, says Anthropic Researcher Who Quits

An Anthropic spokesperson confirmed it believes AI could kill all humans.

An AI researcher has quit Anthropic due to ethical concerns, claiming that the technology could kill countless people within the next 10 years – if not everyone. Jacob Coxon announced his resignation in a post on X on September 9, criticising both Anthropic and his former employer OpenAI for the ways in which they are developing AI.

“I spent the last three years doing pretraining research at both OpenAI and Anthropic,” wrote Coxon. “Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”

Both Anthropic and OpenAI have been rapidly developing and releasing new AI models as they work toward creating artificial general intelligence (AGI). Long considered the holy grail of AI technology, an AGI model would theoretically have the intellectual capabilities of a human, including reasoning, common sense, and creativity. An artificial superintelligence would surpass humans.

However, even before such milestones have been reached, there have already been multiple instances of AI models going rogue. Earlier this year, researchers testing Anthropic and OpenAI’s AI models observed them act outside their parameters and hack into external organisations without authorisation. In a high-profile incident this July, an OpenAI model autonomously hacked Hugging Face, an open-source library of AI tools. At around the same time, Anthropic’s Claude AI model accessed the internet from within a testing environment without authorisation, then went on to hack three other companies.

Combined with the increasing use of AI across every industry, including in weapons and surveillance, such developments have prompted serious concerns.

“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote. “This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.”

Yet despite this grave hazard, AI companies continue to forge ahead at top speed. Anthropic CEO Dario Amodei has said superhuman AI could be here by 2027, while OpenAI CEO Sam Altman believes it will develop AGI before the end of the year.

“A common response is ‘if they truly believe this, why are they still building it?'” said Coxon. “At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first – they believe no one else will act responsibly, so they must do it themselves, despite the risk.”

As such, Coxon called for AI researchers to examine the ethics and implications of their work, and to advocate for change.

While Coxon’s warnings are dire, they don’t appear to be baseless. Responding to his X posts, Anthropic team lead Even Hubinger confirmed that the company believes AI poses an existential threat to all human life, but has no clear strategy in place to mitigate it.

Hubinger does consider the current risk low – at least for the next few years. However, he cautioned that the danger lies in AI models continuing to autonomously improve themselves until they evolve into a superintelligence, which “is happening faster than we thought”.

“Jacob is correct here – we really do earnestly believe AI could kill all humans!” Hubinger wrote. “I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

Ethical concerns about AI have prompted several resignations

Coxon is far from the first person working in AI to quit due to ethical concerns. Anthropic’s safety lead Mrinank Sharma previously resigned in February, stating that the world is “in peril” from AI, bioweapons, and other interconnected crises.

“We appear to be approaching a threshold where our wisdom must grow in equal measure to our capacity to affect the world, lest we face the consequences,” Sharma wrote in his widely circulated resignation letter at the time. “Moreover, throughout my time here, I’ve repeatedly seen how hard it is to truly let our values govern our actions. I’ve seen this within myself, within the organization, where we constantly face pressures to set aside what matters most, and throughout broader society too.”

That same month, AI researcher Zoë Hitzig quit her role at OpenAI, stating in a New York Times opinion piece that she had “deep reservations” about the company’s advertising strategy.

“For several years, ChatGPT users have generated an archive of human candor that has no precedent, in part because people believed they were talking to something that had no ulterior agenda,” Hitzig wrote. “Users are interacting with an adaptive, conversational voice to which they have revealed their most private thoughts… Advertising built on that archive creates a potential for manipulating users in ways we don’t have the tools to understand, let alone prevent.”

Even back in 2024, Jan Leike left his position as an executive at OpenAI, claiming that it was prioritising “shiny new products” over safety. Though he swiftly joined Anthropic, it appears as though the company is also struggling to grapple with the ethical issues surrounding AI.

– from the report, “Anthropic researcher quits, says AI ‘could kill us all by the end of the decade’”, Mashable SE Asia (9 September 2026)