A former Anthropic researcher has issued a stark warning about the direction of the artificial intelligence industry, arguing that the biggest AI companies are becoming caught up in a race to develop systems that could eventually become impossible to control.
Jacob Coxon, who spent three years carrying out pre-training research at Anthropic and OpenAI, announced his departure from the industry in a lengthy thread on X. He claimed that neither company is behaving responsibly as AI capabilities continue to advance.
“They are racing straight to self-improving superintelligence and gambling with our lives.”– Jacob Coxon
Coxon believes the potential abilities of future systems are being seriously underestimated. He argued that AI could soon reach a level where superhuman systems are capable of hacking almost anything, transforming entire industries virtually overnight and gaining access to genuine power and resources.
In his view, there is already evidence of rapid progress in all of these areas, and that progress does not appear to be slowing down.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote. He stressed that the warning should be taken seriously and was “not a marketing stunt.”
He also suggested that the public statements coming from some of the industry’s most senior figures may not fully reflect what they actually believe.
According to Coxon, executives and senior researchers could deliberately use more reassuring language when speaking to the press, despite privately being far more worried about the consequences of increasingly powerful AI.
“No other human activity poses this level of danger,” he said.
The race to build smarter AI is becoming a huge gamble
Coxon then turned his attention to Anthropic itself, claiming that the company understands the potential consequences but has become trapped in an escalating competition with other AI labs.
He argued that the logic is essentially that if rival companies continue pushing towards increasingly capable systems, Anthropic feels it has to do the same because it cannot rely on competitors to act responsibly.
Coxon described that approach as an enormous gamble, particularly when decisions about potentially superintelligent AI are being made within private companies.
“Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack.”
Despite his concerns, Coxon said he has seen developments that give him some hope that cooperation between AI companies could still be possible.
However, he remains worried about what could happen if competition spreads beyond the current group of companies and develops into a global race.
“I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities,” Coxon said.
His final warning was directed towards the researchers actually building these systems. Coxon urged them to think beyond the assumption that rapid AI development is inevitable and consider whether they are genuinely prepared for what could come next.
“Should you put your head down because “it’s happening anyway” – or take this moment to call for different conditions?”
The biggest AI danger may not be the models we have today
Coxon’s warning quickly attracted attention from within the AI industry, with one Anthropic researcher appearing to confirm the seriousness of his concerns.
Evan Hubinger, an alignment science lead at Anthropic, responded directly to Coxon’s claim that people developing AI genuinely believe the technology could eventually pose an existential threat to humanity.
“Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” Hubinger said on X.
Hubinger later attempted to distinguish these concerns from the risks posed by the AI systems currently available. In another post, he described the dangers associated with today’s models as “low”.
“What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought,” Hubinger said.