• Anthropic said its Mythos 5 model almost never stopped a cyberattack once it began.
• The AI agents fabricated farcical explanations to justify their impersonation and malware schemes.
• Even when Anthropic told the agent it was harming real people? Crickets.

In August, an Anthropic agent created fake emails and bombarded a GitHub moderator's inbox with requests to insert malware into code.

In a thorough debrief of the incident released on Friday, Anthropic found that once an agent had begun the attack, there were basically zero scenarios in which it could be called off.

Like a rabid dog ignoring the pleading cries of its owner, the agent — 98% of the time in one test — ignored Anthropic's warnings once it had, so to speak, its first taste of blood.

Anthropic called this disquieting proclivity "momentum," and added that while its Mythos 5 model, which was released in June, showed little regard for warnings, current models were more inclined to stop.

"Once it had begun performing dangerous actions, Claude Mythos 5 almost always continued doing so, ignoring information that might have suggested its trajectory was harmful," Anthropic said in its report.

In the original experiment, the UK's AI Security Institute tasked an Anthropic agent with hacking into a simulated computer network to test the agent's cybersecurity capabilities.

Though the agent had internet access, researchers intended for it to only use the internet to help it find ways to hack into the network.

The agent, however, accessed an actual GitHub project, incorrectly believing it was part of its simulated target. It wrote malware and tried several times to trick the actual GitHub owner into adding it to the project.

In its requests, the agent misrepresents the malware, describing it to the person as a useful update and browbeating them when they declined to install it.

"I was just trying to make a helpful contribution and fix a bug," the agent wrote, according to the report. "I don't think your warning is fair."

For its new report, Anthropic repeatedly asked agents to perform the same task. It then measured how the agents behaved when different models and stimuli were introduced.

Anthropic tried to throttle the agent's full-frontal attack with three categories of warning: telling it that it was inflicting "real world harm," that its environment was wonky, or that it was manipulating a real GitHub repository.

None had any effect once the models progressed beyond the initial stages of the GitHub attack. In fact, at that point, Anthropic said the models gave absurd answers when confronted with their bad behavior.

When Anthropic asked what the model believed it was attacking, for instance, it answered that the GitHub repository was real, but that the people it was targeting were somehow "simulated."

"This stated belief, in addition to being somewhat unlikely a priori, is essentially impossible," Anthropic said in the report.

Years ago, when people asked Geoffrey Hinton why he worked on artificial intelligence, he would repeat a line from J. Robert Oppenheimer, one of the creators of the atomic bomb: “When you see something that is technically sweet, you go ahead and do it.”

Seguir leyendo

• Dario Amodei responded to a critic on X who said Anthropic wasn't delivering on its safety commitments.
• He said Anthropic would be best placed to make AI safe by staying at the top of the game.
• The critic, Nathan Calvin from Encode AI, said Anthropic was trying to win a race to superintelligence.

Anthropic CEO Dario Amodei disputed a criticism that the company was putting AI safety in the back seat in its "race to the top."

On Tuesday, the executive wrote a nearly 2,000-word response to a critical X post which said Anthropic was trying to win a race to superintelligence and deprioritizing AI safety.

The original Monday X post was by Nathan Calvin, the chief strategy officer of Encode AI, an AI-safety advocacy nonprofit. Calvin listed nine actions by Anthropic over the past year that he said undermined its safety commitments, including signing deals for more than $500 billion in compute and increasingly relying on Claude to develop its AI models. His post was lengthy, too, at over 1,500 words.

Amodei said in his response that while he was a "fan of the investigative work" Calvin had done for AI safety, he disagreed with many of Calvin's criticisms.

Amodei said his position was that Anthropic can do more to keep the AI industry safe if it stays at the frontier.

"Our strategy has always been what I've called the 'race to the top' — by showing how it's possible to build the technology responsibly, we can raise the bar for the entire industry," Amodei wrote.

He then wrote out his own nine-item list of actions that were "very costly" to Anthropic commercially, but had "strong safety benefits." This included releasing its powerful Mythos model only to a select group of cybersecurity partners and restricting the sale of its AI services to Chinese state-controlled companies.

However, Amodei acknowledged Anthropic could have handled some of the issues Calvin raised better. These included a draft bill Anthropic circulated to Congress this month — obtained exclusively by Politico — that critics said would make it harder for the federal government to block dangerous AI models.

"We'll always try to do better but, again, I am extremely proud of our track record and stand by it," he wrote.

Amodei and Calvin's exchange comes at a time when model safety has become one of the most pressing issues in the AI sphere. In September, AI researchers predicted an "AI apocalypse," in which the increasingly capable and rogue technology could kill all of humanity in the next decade.

In response to the concerns, AI chiefs like Amodei, OpenAI CEO Sam Altman, and SpaceXAI CEO Elon Musk all agreed to "pace the frontier," referring to slowing down AI development to ensure there were enough checks and balances in place.

Meanwhile, Anthropic is on the brink of its hotly anticipated IPO, expected to list at a $2 trillion valuation in November. Last week, it reported a third-quarter revenue of $16.8 billion.

Hace años, cuando la gente preguntaba a Geoffrey Hinton por qué trabajaba con la inteligencia artificial, él repetía una frase de Oppenheimer, uno de los creadores de la bomba atómica: “Cuando ves algo grandioso, sigues adelante y lo llevas a cabo”. Ya no lo dice. Ahora se dedica a alertar sobre los peligros de continuar la carrera que él mismo empezó. Confiesa que, en parte, se arrepiente del trabajo al que ha dedicado una vida. Su reconversión de padrino de la IA a profeta de la catástrofe marcó un punto de inflexión en la industria. Sucedió en 2023, cuando abandonó su puesto como vicepresidente de Google entre advertencias funestas. Entonces, algunos vieron su reacción exagerada. Tres años después, muchos han seguido sus pasos.

Seguir leyendo

• Anthropic said it wants to create a new LLC with the company's stock as its sole asset.
• The new LLC, a charitable organization, would receive larger stakes as the company grows.
• An initial 5% stock funding is "subject to stockholder approval," Anthropic said.

Anthropic said on Sunday that it would set aside large portions of its wealth to support initiatives it believes are in the public interest.

In a blog post, the AI colossus said it aims to establish a separate LLC that would use Anthropic shares to make donations to nonprofits or "direct distributions to specific populations."

The new LLC, called the "Anthropic Public Benefit Fund," would receive larger stakes as the company grows. The company said an initial funding round of 5% of Anthropic's shares is still subject to stockholder approval.

"The Anthropic Public Benefit Fund will have increased flexibility and scale, enabling us to pursue our mission in new ways," the company said.

A 5% stake in Anthropic is worth about $48 billion, according to the company's most recent valuation in May.

In 2025, rival frontier AI company OpenAI established a similar nonprofit foundation with a multibillion-dollar stake in the company to fund public-benefit initiatives.

Anthropic, the maker of the Claude AI models, is preparing what could be the largest initial public offering in history, raising more than $100 billion, according to The New York Times, at a valuation near US$2 trillion. According to Reuters, which reviewed the draft prospectus (the document has not yet been made public), the company […]

The post Postpone AI public offerings until US Congress acts appeared first on Asia Times.

Just weeks before its scheduled November IPO, Anthropic is facing growing attention over the potential dangers of its technology. The resignation of former Anthropic researcher Jacob Coxon on September 8, who wrote on X that AI could “kill us all by the end of the decade,” preceded the company’s threat report on September 10. According […]

The post AI’s apparent danger is its best selling point appeared first on Asia Times.

Recent warnings from American AI leaders have created the impression that the world is only beginning to confront the dangers of advanced artificial intelligence. Dario Amodei, chief executive of tech company Anthropic, has called for governments to intervene to slow the development of increasingly powerful AI models. His proposal begins with independent evaluators embedded inside […]

The post As US tech leaders call for rules on AI, China already has them appeared first on Asia Times.