Politico: The new race to keep up with AI
By AARON MAK | 05/20/2026 04:03 PM EDT
Anthropic recently reported that its AI is showing early signs of recursive self-improvement. | Don Feria/AP
In what could be the first step toward the AI apocalypse — or a techutopia, depending on who you ask — the technology is purportedly on the precipice of autonomously improving itself without human intervention.
Taken to the extreme, recursive self-improvement, or RSI, could lead to artificial intelligence that autonomously advances to the point at which humans are no longer able to control it — a theoretical milestone known as the singularity. That could cause the technology to wipe out humanity, or usher in the kind of leisurely world without work that Elon Musk has envisioned. There are plenty of less dramatic outcomes for RSI, though, which may arguably be more likely. In any case, it’s unclear if society is ready to handle the impact after Anthropic disclosed early signs of RSI in its AI earlier this month.
If humanity is already struggling with the technology right now, how can we hope to regulate this latest twist? Existing and proposed policies for current AI — like auditing its code, prohibiting certain uses (think nuclear weapons) or requiring government approval before releasing a model — may be inadequate if the technology is autonomously advancing faster than humans can react. Some legal and computer science researchers told DFD that RSI will force us to rethink what viable AI regulation looks like.
“When you are going into RSI, you are setting yourself up to have a system that can iterate faster than anybody can track,” said Eran Kahana, an attorney and AI research fellow at Stanford Law School. “RSI [has the] ability to self-iterate and basically become something that is unrecognizable.”
One solution could be to set up liability rules early. It’s hard to predict what harm RSI might cause, but you could imagine a scenario in which a model upgrades itself enough to evade safeguards and cause a biohazard disaster. In such an event, there would need to be a way to figure out who is responsible for any deaths and injuries.
The liability framework could take several forms.
An industry-friendly approach would be to focus on whether the developer should have reasonably foreseen the harm caused by their model. This regime addresses the fact that RSI may cause AI to evolve in unexpected ways, and a developer becomes less and less liable the more the model changes from its original state. The upside of this regime is that it would promote innovation, yet it could also make it harder for victims to get justice.
A safety-focused approach could impose strict liability, which would hold developers responsible regardless of whether they expected a model to iterate in a certain way. It’s the same standard we use for accidents at nuclear plants and for oil spills.
“Of course, then there will be the people who say [that] you can’t do that because then nobody will want to develop [RSI],” said Kahana. “Maybe, maybe not.”
Similarly, Kahana suggests that there could be a statute that automatically imposes, say, a $10 million fine if any harm occurs.
An even better vehicle might be permits. In other words, the government could require permits for companies to develop RSI, through which they agree to security measures, liability insurance and whatever audits regulators deem appropriate for RSI. A permitting system might seem onerous, but Kahana contends it could be merited in the case of RSI. “I see a permit being a very viable approach,” he told DFD. “The [higher] the risk is of an application, the more amenable it is to a permit structure.”
Perhaps the worst case scenario would be RSI causing models to keep enhancing themselves to the point that humans no longer have control, which allows the technology to set off a civilization-ending disaster. In that case, regulation becomes moot.
The bluntest tool for preventing this would be to enact a moratorium on the development of AI before RSI is achieved, a proposal that David Krueger, an assistant professor of machine learning at the University of Montreal, is pushing through his activism nonprofit Evitable.
“We’re about to push the button, and if the AI isn’t already aligned enough, and if the way we’ve set it up isn’t right, then we’re going to lose control of this thing,” Krueger told DFD. “The obvious policy here is just to slam the brakes.”
Yet Yu Su, assistant professor of computer science at Ohio State University, who also serves as CEO of the AI agent company NeoCognition, is skeptical that RSI will achieve what its biggest detractors predict.
“The very important distinction between these artificial entities and the biological beings is truly innate goals, survival instincts and this whole suite of traits,” he told DFD. “These traits are necessary for some being to escape our control or bring about so-called existential risks, but currently, I don’t see any plausible way to give [those] traits to AI.”
