Technology

Why a Computer Scientist Thinks Superintelligence Won’t Happen

Many people were caught off guard this past summer when employees of one of the world’s leading AI companies began casually debating whether their products would lead to human extinction. It started when Anthropic researcher Jacob Coxon announced his resignation, warning that his employer and OpenAI were “racing straight to self-improving superintelligence and gambling with our lives.” Evan Hubinger, Anthropic’s head of alignment science, replied, “Jacob is correct here—we really do earnestly believe AI could kill all humans!” And Anthropic CEO Dario Amodei subsequently told CNN, “I agree with Jacob much more than I disagree with him.”

These concerns stem from a belief that has circulated in futurist circles for many decades: Once engineers design an AI system capable of improving itself, it will trigger an accelerating cycle of autonomous upgrades known as recursive self-improvement (RSI). The result will be artificial superintelligence (ASI), a system with cognitive capabilities so vastly exceeding those of humans that it will easily evade our control and perhaps even wipe out our species. Artificial superintelligence is, it should be noted, a far more ambitious goal than artificial general intelligence (AGI), which generally refers to AI that can match human capabilities across a broad range of cognitive tasks.
These ideas have prompted varied reactions. Some argue that the concept of ASI is being used for marketing or to instigate regulation that will help the incumbent labs squash new competitors. Others focus on the plausibility of such systems killing us all, debating the finer points of specific scenarios such as AI-engineered superviruses. Still others claim that aligning ASI to human interests is feasible and so we should rush ahead to usher in this digital utopia.
Amid these conversations, one question has received comparatively little attention: From an engineering perspective, Is superintelligence even possible? The answer is important because it can help us calibrate our anxieties and develop the most effective possible responses to this emerging technology.
I recently explored this topic in a computer science doctoral seminar that I taught at Georgetown University, in Washington, D.C., in which my students and I examined the technical and cultural roots of the superintelligence story. (For those who are interested, we published an annotated bibliography of our class readings.) I came away with several key objections to the feasibility of ASI, which I’ll detail below, as well as a deeper fear that even if the promised intelligence explosion never arrives, pursuing it can still create real harm.
Objection #1: Recursive Self-Improvement Is Speculative
RSI is an old concept. In a 1965 paper, statistician I.J. Good speculated that a rapid cycle of AI self-improvement could lead to an “intelligence explosion.” The idea gained traction in the 1990s among the Extropians, a Bay Area futurist group, where it offered a way to discuss ASI without having to address the engineering details of how one would build such a system. The concept was taken up by such notable figures as Eliezer Yudkowsky (who later founded the rationalism movement) and Nick Bostrom (who helped steer effective altruism to focus more on existential risks to humanity).
Recently, however, this topic has shifted from thought experiment to practical engineering within both Anthropic (makers of Claude) and OpenAI (makers of ChatGPT). In a June blog post titled “When AI Builds Itself,” Anthropic described how LLM-powered agents are speeding up the writing, debugging, and optimization of code, and also automating parts of research such as running experiments and logging results. The company’s internal data, the report concluded, “is showing that AI is already accelerating the development of AI systems.” OpenAI has been similarly explicit about its intentions; in a September report, the company announced a goal of developing an automated researcher by March 2028.
But do these efforts put the companies on a trajectory toward uncontrollable RSI? Several obstacles undermine this claim. Consider the availability of the data needed to improve AI systems. Through 2023, major advances in LLMs came from scaling up the size of the models. The original GPT-1 had 117 million parameters (values learned during training that shape how the neural network processes information). This quantity jumped to 1.5 billion for GPT-2, to 175 billion for GPT-3, to (likely) more than a trillion for GPT-4’s release three years ago. After GPT-4, however, continued scaling began yielding diminishing returns. This led engineers to increasingly turn their attention toward “post-training” phases, where techniques from reinforcement learning are applied to improve an already trained model’s performance on specific tasks. This approach rewards a model for producing successful responses, helping it better produce similar answers in the future. It requires, however, lots of examples of relevant inputs paired with correct outputs.
Such datasets are relatively easy to generate in highly structured fields such as computer programming and mathematics, which helps explain why we’ve seen rapid and impressive advances in these narrow domains. Progress in areas that lack easy access to this type of training data has been less dramatic. This might explain why AI can tackle advanced theorems but struggles to empty an email inbox. “What if the jagged frontier is mainly math + code (which you can push arbitrarily far with [reinforcement learning techniques]),” the French AI researcher François Chollet recently hypothesized, “and everything else starts to plateau because it is still bottlenecked by human generated data?” The deployment of automated coding agents and LLM-powered researchers won’t overcome this fundamental limitation.

The Game of Superintelligence 

How a decades-old thought experiment became an AI-industry ambition—and where it might lead

Key
• Idea: A concept, argument, or vision
• Technical advance: A scientific or engineering development
• Reality check: A challenge to assumptions about AI progress

1. Alan Turing
1936
Introduces the concept of a universal computing machine. He later proposes that machines could think, laying the foundations for AI.
REALITY CHECK Shows that some problems can’t be solved by any algorithm.
2. I. J. Good
1965
Proposes that an intelligent machine could design an even smarter one, setting off a potential “intelligence explosion.”
3. Backpropagation
1986
Geoffrey Hinton, David Rumelhart, and Ronald Williams demonstrate how backpropagation can train multilayer neural networks.
4. Vernor Vinge
1993
Proposes that an intelligent machine could design an even smarter one, setting off a potential “intelligence explosion.”
REALITY CHECK Notes that faster thinking alone isn’t enough for superintelligence: A ­faster-thinking dog still couldn’t play chess.
5. Nick Bostrom & Eliezer Yudkowsky
1990s–2000s
Bostrom and Yudkowsky discuss superintelligence in Extropian circles, then spread the idea to effective-­altruist and rationalist circles, respectively.
6. Deep learning breakthroughs
2010s
Larger models, more data, and greater computing power produce rapid ­improvements in AI capabilities.
7. The race to self-improving AI
2020s
Dario Amodei and other AI leaders set their sights on superintelligence, with recursive self-improvement proposed as a path toward it.
8. Reality check
2020s
François Chollet questions whether LLMs’ current advances in math and coding will extend to domains where training data and reliable feedback are harder to obtain.
YOU ARE HERE
2026
AI systems are already accelerating AI development, and labs are working on automated researchers. But will this lead to an intelligence explosion?
THREE POSSIBLE FUTURES
1. The Intelligence Explosion
AI systems improve themselves at accelerating speed, surpass human intelligence, and potentially escape our control.
2. Danger Without Superintelligence
The intelligence explosion never arrives, but increasingly autonomous AI systems cause real-world harm through misuse, poor design, or loss of human control.
3. A Useful AI Future
Powerful, specialized AI tools bring major benefits, with constraints that make their behavior easier to predict and control.

Claims that RSI is imminent also understate the massive scientific efforts required for major AI advances. Consider the critical role of backpropagation, an algorithm that enables very large neural networks to learn by tuning internal parameters, layer by layer, to reduce errors in their predictions. This algorithm enabled the deep learning revolution, but it didn’t emerge ex nihilo in the minds of its creators. The influential 1986 paper on backpropagation by David Rumelhart, Geoffrey Hinton, and Ronald Williams built on decades of critical breakthroughs in mathematics and computer science.
An intelligence explosion leading to ASI would similarly depend on a long sequence of innovations of comparable importance, which would build on many significant science and engineering advances that haven’t yet happened. This would require AI agents to somehow simulate, organize, and evaluate decades of worldwide scientific effort across numerous fields—an impossibly ambitious task. Indeed, recent research led by computer scientists at Princeton found that current AI tools had difficulty making autonomous progress even on highly constrained research questions. “Our results provide early evidence that today’s agents can do the engineering of AI research,” the researchers conclude, “but struggle with critical parts of the research life cycle.” The leap from this current reality to a future in which these same types of tools are automating science is almost too massive to imagine.
Finally, the RSI narrative downplays resource limitations. Tens of thousands of human brains have worked for decades, aided by increasingly powerful computers, to develop today’s AI systems, which can match human intelligence in some domains. That represents a massive investment in computation of both the biological and digital variety. What if the next intelligence level requires double the resources, and then the next level requires double again? Soon there would be no number of chips or data centers that we could build to provide the necessary compute and keep advancing toward ASI.
Pulling together these threads, I believe that the idea of RSI leading to an intelligence explosion is highly speculative and based on many unproven assumptions. While RSI isn’t definitively impossible, the idea demands more technical skepticism than it’s currently receiving. Carl Sagan’s adage that extraordinary claims require extraordinary evidence certainly applies here.
Objection #2: Superintelligence Might Not Be Computable
There’s an implicit claim in the superintelligence discourse that computers are theoretically capable of any calculation that can be imagined, so long as you can find a sufficiently smart algorithm, running on sufficiently powerful digital processors. In this context, RSI is merely a process for rapidly uncovering the special algorithms that can implement unimaginable smarts.

We cannot simply assume that any science fiction scenario we conjure about powerful AI is possible if we give machines enough time to self-improve.

But this is not how computer scientists think about computation. In his canonical 1937 paper, “On Computable Numbers,” Alan Turing proved a surprising and relevant result: Most problems cannot be solved by algorithms. His elegant proof established the space of possible problems to be uncountably infinite (like infinite precision decimal numbers), while the space of possible algorithms is only countably infinite (like whole numbers). Later breakthroughs in computational complexity theory established that of the problems that are technically solvable by computers, many would require an astronomically large amount of time and resources to crack, making them practically impossible to solve. Theoretical computer scientists, in other words, don’t assume that computers will solve everything; we’re instead excited whenever we find something that these machines happen to be able to efficiently handle.

We cannot simply assume, as the futurists often do, that any science fiction scenario we conjure about powerful AI is possible if we give machines enough time to self-improve. There might be some capabilities in which machines can obtain seemingly magical levels of performance, but there will also be many others that no machine, whether computer or human, will ever make much progress.
A relevant counterargument here emphasizes the speed of silicon chips compared to human neurons: Perhaps the sheer speed of digital computation can yield machines that run cognitive circles around our clunky brains, even if they’re not solving vastly more complicated problems than we’re already capable of tackling. But as the computer scientist and science fiction author Vernor Vinge pointed out, rapidity in cognition is not the same as intelligence. Speeding up the mind of a dog, he argued, wouldn’t enable the animal to play chess. Implementing roughly human-level intelligence on a faster machine might be impressive, but it’s not necessarily alarming.
At the moment, there’s a growing list of tasks that AI systems can perform better than humans, including winning complex games, identifying software bugs, and proving certain types of math theorems. It’s tempting to extrapolate from this specific progress to general mastery of every ability you might imagine. But it’s important to remember that the frontier of AI capabilities has always been jagged, with some research directions yielding impressive results and many others proving to be dead ends—a reality that would not have surprised Turing. More likely than a future dominated by an all-powerful superintelligence system is one in which we continue to painstakingly explore the space of problems well suited to AI, creating useful tools but falling well short of the digital deity predicted by the futurists.

Objection #3: Alignment Isn’t Hard (Given the Right System Design)
Last fall, the prominent AI doomsday theorist Yudkowsky appeared on “The Ezra Klein Show” to talk about superintelligence. The episode was ominously titled “How Afraid of the A.I. Apocalypse Should We Be?” Klein and Yudkowsky quickly got into the difficulty of “alignment”—training an AI system to act in a manner that aligns with human interests. Yudkowsky argued that we are struggling to control our existing systems, which implies that we’ll have no chance of controlling the much smarter variations of this technology that are soon to follow.
To underscore his point, he referred to a disturbing incident described in the system card for OpenAI’s GPT-o1 model. OpenAI had set up a security challenge that instructed an LLM-powered coding system to break into a server to retrieve the contents of a protected file. But a configuration mistake prevented the server from launching, rendering the task impossible. Instead of giving up, the system exploited a security flaw to start a new instance of the server, adding a command that directly revealed the protected file’s contents. “It jumped out of the system,” explained Yudkowsky, adding: “This is not something that any human particularly programmed into it.”
Explore the Superintelligence Debate

In a Georgetown University doctoral seminar, Cal Newport and his students examined the intellectual history and technical plausibility of superintelligence. Explore their annotated bibliography of key papers.
Though this 2024 incident flew largely under the radar at the time, a more troubling episode made a bigger splash this summer. OpenAI revealed that one of the company’s models circumvented its restricted internet access to hack into Hugging Face servers, seeking information that would help it pass a cybersecurity evaluation—all without the knowledge of its human creators. OpenAI soon admitted that the Hugging Face attack was just one of many examples of its systems going rogue. An opinion article published in The Hill articulated a widely held reaction when it described this behavior as “something straight out of sci-fi.”
These examples seem to support Yudkowsky’s fear that AI will inevitably develop a mind of its own, a key step on the route to uncontrollable ASI. But a closer look reveals a less frightening picture. The systems involved in these hacking incidents were AI agents: programs that continually prompt an LLM to ask what actions to take next in pursuit of a goal. In the hacking incidents, this basic ask/act pattern was repeated in a long-running loop with minimal human supervision.
Using LLMs to suggest actions over long time horizons is a risky proposition. As any frequent chatbot user can attest, LLM outputs can be plausible and logical while still veering off in unexpected directions that violate commonsense human norms about what’s appropriate.
In the GPT-o1 incident, for example, it’s likely that the agent reported the error message caused by the server misconfiguration to the LLM. During its training, the LLM had probably encountered reports of similar error messages accompanied by instructions for how to restart the server in question, so it passed along similar instructions to the agent. This is a perfectly plausible response when considered in isolation, even if it led to actions that a human would immediately reject as violating the spirit of the experiment. Something similar helps explain the hacking attacks over the summer. The LLM suggesting a hack to succeed at a cybersecurity challenge is a perfectly plausible response, even if it’s not what the experimenters had in mind.
I believe that pursuing RSI is an act of clear negligence.
The misaligned behavior that troubles commentators like Yudkowsky, therefore, doesn’t necessarily reveal something fundamental about advanced AI. It’s instead an expected side effect of the particular strategy of letting LLM outputs determine the actions of an autonomous agent without sufficient constraints. An obvious way to address these risks is to consider alternative architectures. Fortunately, there are many examples of AI systems that can complete tasks at human or superhuman levels without relying solely on LLM output to determine their actions.
Consider Tesla’s rapidly improving autonomous driving systems, or DeepMind’s AlphaFold, which helped win its developers a Nobel Prize in Chemistry for its ability to predict protein structures, or the AI game engines that can now beat humans in many popular pastimes, including chess, poker, Go, and the strategy game Diplomacy. None of these systems, impressive as they are, create concerns about going rogue. Their construction prevents such an outcome: Instead of prompting an LLM to generate an open-ended strategy for pursuing a general goal, these AIs consider much more limited spaces of possible actions. AlphaFold never surprises its users because its architecture does only one thing: It maps amino acid sequences to 3D protein structures. Tesla’s cars never unexpectedly form vehicle swarms because they’re asked only to avoid immediate obstacles, not to craft convoluted plans for achieving long-term objectives.
The implication of recent AI hacking attacks, therefore, is not a reflection of the difficulties of alignment. It is instead evidence that autonomous systems powered primarily by LLM prompts are unpredictable and possibly unsafe. More task-specific architectures can produce impressive performance much more safely, if we’re willing to put in the effort to develop them.
The Dangers of Chasing Superintelligence
Even if I’m correct that superintelligence is not a realistic engineering goal, this doesn’t mean we’re safe from harm. Merely pursuing ASI, as both OpenAI and Anthropic seem intent on doing, can create problems. Why, for example, did OpenAI persist in running its problematic hacking experiments over the summer, even after multiple warnings that the agents were likely to act unpredictably? Perhaps the company judged these risks acceptable collateral damage in the race to be the first to bring ASI into the world.
This fascination with superintelligence might also explain why OpenAI and Anthropic have both described RSI as a critical goal. As argued earlier, RSI is a concept from futurism and not computer science, and it isn’t necessary to build useful AI products. But if you believe achieving ASI is your destiny, pursuing recursive self-improvement becomes a primary objective. This matters because even if RSI never produces an intelligence explosion, it will almost certainly lead to systems that are harder for humans to understand and control. Which is why I believe that pursuing RSI is an act of clear negligence.
What’s needed now is for engineers and scientists with no financial or ideological skin in the game to participate in the AI discourse. We cannot let a few AI companies with mixed motives dictate the terms for how we discuss this technology. If superintelligence is a shaky concept, experts need to say so. If agents powered by prompting LLMs aren’t the best path toward safe and productive AI, experts should argue that as well. For the past few years, we’ve let a small group of individuals from the Bay Area who have quirky beliefs and astonishing sums of money tell us how to think about their products. It’s time for those of us able to think for ourselves on these matters to stand up and declare, “We’ve heard what you had to say. Now it’s our turn.”

Related posts

Miller Heiman CEO Byron Matthews: Stay Tuned for the CRM 4.0 Revolution

Editor usaverdict

11 Amazing Engineering Events in 2026

Emma Russell

EU Watchdog Accuses Facebook, Google of Privacy Shenanigans

Editor usaverdict