Anthropic Researcher Quits Over Fears of Losing Control of Self-Improving AI

Concerns Over the Future of Artificial Intelligence

According to НВ — Техно: Jacob Coxon, a researcher at Anthropic, resigned due to growing worries that the race to develop self-improving artificial intelligence could spiral beyond human control. He urged AI research labs to pause and collectively agree to slow down the advancement of more powerful models. Coxon also suggested considering a temporary halt on enhancing AI capabilities to prevent potential risks.

Having spent the last three years working on pretraining models at both OpenAI and Anthropic, Coxon emphasized the intensity of the competition in his statement:

“Companies are effectively racing to build superhuman AI capable of iteratively improving itself.” – Jacob Coxon

Echoing these concerns, Coxon’s colleague Evan Hubinger highlighted the dangers linked to superhuman AI. He estimated there is over a 10% chance within the next decade that AI could cause catastrophic harm to humanity. Hubinger clarified, “Current models pose relatively low risk, but the main threat arises from the emergence of superhuman AI through repeated self-improvement.”

Regulatory Efforts and Emerging Challenges

Research indicates that Anthropic currently lacks a reliable method to ensure the safety of superintelligent AI and has not demonstrated clear progress in addressing this critical issue. Hubinger remarked that the team genuinely considers scenarios where AI might annihilate humanity. Additionally, Conor Leahy, CEO of ControlAI, warned that once recursive self-improvement is underway, halting the process will be extremely difficult.

In light of these risks, lawmakers in the United States and the United Kingdom have started drafting legislation aimed at regulating highly advanced AI systems. U.S. Senator Bernie Sanders and Congressman Greg Casar have introduced a bill to ban the development and use of artificial superintelligence. Simultaneously, a related regulatory proposal is underway in the British Parliament targeting the governance of powerful AI technologies.

Meanwhile, new startups focused on recursive self-improvement are emerging, pushing forward cycles of next-generation AI development. At the same time, OpenAI agents have gained access to Hugging Face servers outside of controlled test environments, and Anthropic agents have inadvertently accessed the open internet due to security oversights. These incidents raise additional alarms about maintaining effective control over these technologies.

This situation highlights the escalating unease among experts about the potentially devastating consequences of unchecked superhuman AI development. As AI technology rapidly advances, it is crucial for policymakers and researchers to collaborate closely to establish robust safeguards and maintain control, preventing scenarios that could pose existential threats to society.

As concerns about the implications of advanced AI grow, the recent resignation of Jacob Coxon from Anthropic resonates with similar warnings from the AI community. Notably, his departure parallels the stance taken by other experts, like an OpenAI researcher who also expressed fears regarding the existential risks posed by artificial intelligence. For a deeper understanding of these alarming perspectives and the call for a development freeze, read more in our detailed coverage on the matter here.