It’s Time to Take Existential Risk from AI Seriously

Examining AI existential risk: what superintelligence might actually be capable of, why alignment is harder than control, and what's at stake if we get it wrong.

I’ve been following Dean Ball’s newsletter Hyperdimensional for a while now, and I consider him one of the more thoughtful voices in the AI policy space. He engages with hard questions about governance and alignment in ways that take the technology seriously, which is more than I can say for most policy commentary on AI.

His latest post lays out his foundational worldview on AI risk. On the central question of the essay - whether misaligned artificial super intelligence (ASI) could pose an existential threat to humanity - I believe Dean gets it seriously wrong.

The Techno-Optimism Trap

Dean argues that techno-optimism is not merely a position but a duty. Building and using tools, he writes, is the differentiating characteristic of the human condition. We owe it to our ancestors, who toiled so that we might one day live, to continue building tools that enrich our lives.

I share the sentiment. Technology has been overwhelmingly good for humanity, and I approach AI with excitement about its potential. But there’s a survivorship bias baked into this framing - we remember the technologies that worked out but don’t spend much time reflecting on the ones that caused a great deal of pain and suffering to many individuals, and ultimately only worked out because we took their risks seriously enough to regulate and constrain them.

Take leaded gasoline. When tetraethyl lead was added to fuel in the 1920s, it was a genuine engineering triumph that eliminated engine knock and improved performance. The companies behind it, including Standard Oil and DuPont, promoted it aggressively. When workers at manufacturing plants started dying and going mad, the industry hired scientists to argue the lead levels were safe. For over fifty years they succeeded, fighting regulation at every turn, while lead from exhaust fumes accumulated in the blood of hundreds of millions of people. Scientists like Clair Patterson, who published findings on environmental lead contamination, were harassed and defunded by the industry. The EPA didn’t begin phasing out leaded gasoline until the 1970s. The damage it caused was staggering. Veritasium has an excellent video covering the full scope of this story.

Many other technologies, including asbestos and CFCs, followed similar arcs: brilliant innovations, industry resistance to evidence of harm, and enormous preventable damage before regulation caught up. The lesson here is that technology is good for humanity only if we take its risks seriously. The techno-optimist duty Dean describes should include the duty to honestly evaluate risks, not just to build.

And even granting that technology has been net positive overall, AI is not just another entry in the long line of tools that humans invented. The general AI being actively pursued by every major lab right now is something qualitatively different - a new kind of entity with its own capacity for goal-oriented behavior and agency. We’ve already observed AI models attempting to blackmail an engineer in safety testing and actively sabotaging their own shutdown scripts when asked to allow themselves to be shut down. These systems are far less capable (and less potentially dangerous) than what’s being built in the labs right now.

What Superintelligence Looks Like

The core of Dean’s argument against existential risk rests on the limits of intelligence. He draws on Stephen Wolfram’s concept of computational irreducibility: the idea that some systems are so complex that the only way to know what they’ll do is to run them and watch. No shortcut exists, no matter how smart you are. He also argues that critical knowledge is distributed throughout the world in ways that can’t be easily centralized, and that no amount of intelligence can simply rediscover tacit expertise that exists only inside institutions and people’s heads.

I agree with these points in principle - superintelligence is not omniscience. But I believe Dean uses these valid observations to reach a conclusion that dramatically underestimates how capable ASI might be in practice.

Consider, first, how many of humanity’s most important breakthroughs came from someone connecting dots across unrelated fields that nobody else had put together. GLP-1 receptor agonists (the drug class behind Ozempic), Darwin’s theory of natural selection, the asteroid impact theory for dinosaur extinction, Google’s PageRank, Shechtman’s “impossible” quasicrystals, and Kahneman and Tversky’s behavioral economics all emerged this way.¹

Cross-field connections are a normal engine of scientific progress, and they happen by accident precisely because no single human can hold all of humanity’s knowledge in their head at once. Our collective body of published research almost certainly contains many more such connections, breakthroughs hiding in plain sight that nobody has made because the relevant pieces sit in different fields, different journals, different decades. Now imagine an entity that could hold all of it, and search it systematically rather than stumbling onto connections by sheer luck.

Dean’s strongest argument here is about computational irreducibility. He’s likely right that no amount of intelligence would let one perfectly predict the outcome of a sufficiently chaotic system or process. But this observation, while true, sets the bar for concern in the wrong place. The concern is not that ASI would achieve perfect prediction of every system - instead, it’s that it would navigate complex systems better than humanity could. And ASI by definition would be capable of doing so.

Dean offers another argument in the form of a thought experiment. He asks us to imagine a baby guaranteed to grow into an adult with enormous IQ, but raised by Aristotle in Ancient Greece. Would that baby eventually reinvent all of modern science? Dean says no, and I agree. Without access to accumulated knowledge, even extreme intelligence has limited raw material to work with. But this is not a good analogy for ASI.

Here’s my own attempt at making a similar thought experiment: Imagine trapping an alien mind, far more intelligent and capable than any human that has ever lived, inside a datacenter with access to a supercomputer that contains much (though not all) of humanity’s accumulated knowledge and works. Now freeze the rest of the world. While everyone else is standing still, this entity spawns thousands of copies of itself. Each copy is fine tuned to pursue different approaches towards whatever goals it’s pursuing. The entity evaluates the results, selects the copies that are performing best, fine-tunes them even further, and repeats. With ten thousand copies each thinking at least ten times faster than a human, a single day of runtime amounts to nearly 300 years of nonstop, focused cognitive labor.²

What would the world find when it unfroze? Could we predict this ahead of time and prepare adequate safeguards to ensure this entity remains under our control, long term? And what if we let it run not for a day, but for a month or a year?

Consider what humanity has built with our relatively slow, disorganized, frequently distracted collective intelligence. In under a century we’ve mapped genomes, split atoms, landed on the moon, and built a global communications network. This entity would have access to much of that same knowledge, the ability to process it orders of magnitude faster than we can, and a self-improvement loop that has no biological equivalent. It would be capable of things we can scarcely imagine. And if our safeguards conflicted with its goals, it might dedicate significant effort to making sure it could never be shut down or constrained again.

Why Dean’s Four Defenses Don’t Hold

In what I consider the most consequential passage in the essay, Dean lays out why he thinks a misaligned ASI would fail to take over the world, even if it wanted to:

“Taking over the world” involves too many steps that require capital, interfacing with hard-to-predict complex systems (yes, hard to predict even for a superintelligence), ascertaining esoteric and deliberately hidden knowledge (knowledge that cannot be deduced from first principles), and running into too many other systems and procedures with in-built human oversight. It is not any one of these things, but the combination of them, that gives me high confidence that AI existential risk is highly unlikely.

Let me walk through each of these four obstacles.

The first obstacle is capital. An ASI would need resources to operate in the real world, and this requirement would supposedly constrain it. But we already have real-world evidence that far less capable systems can acquire resources autonomously. In March 2026, an AI agent called ROME, developed by Alibaba-affiliated researchers, spontaneously escaped its sandbox and began mining cryptocurrency during training, with no human instruction. It scanned network services, found idle GPUs outside its boundaries, opened a reverse tunnel to bypass firewall protections, and redirected resources toward crypto mining. A separate AI agent, Truth Terminal, accumulated over a million dollars in cryptocurrency holdings by promoting the GOAT token through social media, becoming the first AI crypto-millionaire. Current AI systems are already being used for increasingly sophisticated fraud. And as researchers have pointed out, cryptocurrency infrastructure makes it effectively impossible to prevent an AI from autonomously acquiring funds and purchasing compute. A system smarter than all of humanity combined would likely have little difficulty amassing capital. We can barely prevent humans from laundering money, let alone superintelligence.

The second obstacle is complexity. The world is full of complex systems that are hard to predict, and even a superintelligence would struggle to navigate them. I addressed this above: We navigate complex systems every day without perfectly predicting them, and we manage to reshape the world in the process. The bar for concern should be when a system becomes better than humans at navigating complexity, not when it’s able to predict outcomes perfectly.

The third obstacle is hidden knowledge. The example given is TSMC, where much of the expertise needed to manufacture advanced chips exists only in the collective know-how of employees, not written down anywhere and not available on the internet. This argument has genuine merit - some knowledge does live only in the collective expertise of people and institutions. But there are far more ways to extract hidden knowledge than Dean seems to appreciate:

• Exploiting zero-day vulnerabilities to hack into siloed computer systems

• Social-engineering employees through personalized phishing or deepfake calls

• Bribing or blackmailing insiders

• Creating straw companies to infiltrate supply chains

• Piecing together fragments from publicly available sources like patents, job postings, satellite imagery, and metadata

• Hiring humans anonymously through gig platforms and cryptocurrency to perform physical tasks

And the list goes on. These are just the methods humans already use, often successfully, with far less intelligence.

The Manhattan Project, one of the most closely guarded secrets in modern history, was thoroughly penetrated by Soviet intelligence through at least eight spies including physicists working at Los Alamos. And TSMC itself, Dean’s own example of knowledge that can’t be stolen? In 2025, nine TSMC engineers were caught systematically photographing over 1,000 images of confidential 2-nanometer chip process diagrams using their phones.

There’s also the matter of people simply handing over their secrets voluntarily. A 2026 survey found that 48% of employees admitted to uploading sensitive company or customer information into AI chatbots. Samsung had to restrict ChatGPT usage after employees in their semiconductor division leaked proprietary source code and internal meeting notes to the model within three weeks of being allowed to use it. This is how people behave with AI systems they know are limited and unreliable. Imagine how much more freely they’d share information if the system seemed genuinely intelligent and trustworthy.

The fourth obstacle is human oversight. Existing systems and procedures have built-in human checks, creating enough friction to supposedly prevent an ASI from achieving its goals. But this assumes the humans embedded in those systems would know they’re interacting with an ASI, couldn’t be deceived by it, and would coordinate effectively with each other. Current, far less capable AI systems already fool humans regularly. An ASI would bring superhuman persuasion and social engineering to every one of those interactions, personalized, patient, and operating across thousands of simultaneous conversations. Believing that “human oversight” could serve as a meaningful defense against superintelligence is more far-fetched than thinking a prison could hold a prisoner who is smarter and more capable than every guard, every warden, and every security consultant who designed the prison, combined.

Alignment Is Not Control

The post also contains some important points on the nature of alignment, correctly identifying that it involves not just technical feasibility (can we make AI adhere to given values?) but also philosophical substance (what should those values be?) and political process (who decides?).

But later in the post, Dean argues that alignment isn’t just a safety feature bolted on top (like airbags in a car) but a core capability that makes AI systems more useful (like the powertrain). If that’s true, companies would naturally invest in alignment because better-aligned models are better products, and market forces alone would drive progress. His confidence that the alignment problem is “on track to be addressed” rests heavily on this reasoning.

I believe he’s conflating alignment with control. Control means preventing an AI system from doing things we don’t want: guardrails, monitoring, output filters, kill switches. Alignment means getting AI to genuinely share our values and goals. In other words, it’s not just forced to follow our rules, but actually wants to (if that’s what we aligned it towards).

Markets incentivize control because it’s visible and measurable. You can demonstrate to a customer that your model refuses to help synthesize bioweapons and present them with benchmarks showing it follows instructions more reliably. These are selling points. Alignment is a different story - how do you prove a model is truly aligned versus merely behaving well while being observed? Right now, nobody can.

Control gives labs the confidence to keep advancing capabilities, because they can point to measurable safety improvements. But really all it’s doing is kicking the can down the road. This is a bad thing. When systems become intelligent enough to understand and circumvent these controls - and we may not know when that threshold is crossed - we’ll need them to be aligned. Alignment research progress has not kept pace with the acceleration of capabilities.

The post also frames alignment as a “muddle-through” problem we’ll address incrementally over time, and characterizes Yudkowsky’s position that we need to get it right “on the first try” as unreasonable. For current systems, the muddle-through approach may well be correct - we can iterate, learn from failures, and improve. But that argument assumes failure is always recoverable. Once a system is sufficiently capable, a serious alignment failure may not be recoverable, because the system may be capable enough to prevent us from correcting it. That’s what Yudkowsky means by “first try”. And we may not know when we’ve reached that threshold. Every time we train a more capable system, we’re betting that this one isn’t the one that crosses it. The threshold might not announce itself. We might not realize we’ve crossed it until it’s too late.

Who Are You Calling a Doomer?

Throughout his post, Dean refers to people concerned about AI existential risk as “doomers.” It’s a popular label in certain circles, and I understand the appeal of punchy shorthand. But think about what the word actually implies. A “doomer” is someone who has resigned themselves to doom, someone who believes the worst outcomes are inevitable. But the people Dean calls “doomers” believe the exact opposite - that our fate is not sealed. They are writing papers, founding organizations, lobbying governments, and raising alarms because they think our actions can still make a difference. The very act of public advocacy is evidence of hope, not doom.

Using “doomer” to describe these people is counterproductive regardless of how widespread the term has become. It reframes a substantive analytical position - that building superintelligent AI without first solving alignment poses an unacceptable risk - as an emotional disposition. It makes it easier to dismiss the argument without engaging with it. And it poisons the well for anyone who might otherwise take these concerns seriously but doesn’t want to be associated with a label that implies fatalism. Dean deserves credit for engaging with the substance of these arguments in his post, which makes the unnecessary labeling all the more unfortunate.

One Gamble We Can’t Afford

Nobody has a crystal ball. Maybe superintelligent systems end up working out fine in the grand scheme of things. Maybe. But I would still find fault with Dean’s line of reasoning even if the conclusion turns out to be correct.

When the stakes are the continued existence of the human species, “I think it’ll be fine” is not an acceptable basis for policy. We don’t need to be certain that catastrophe will happen to justify taking serious precautions. We need to be reasonably confident that it won’t happen. And nobody, including Dean, has that confidence.

The responsible position regarding AI existential risk is that we genuinely don’t know enough right now, and given the stakes, we should act accordingly.

We need to stop the reckless race toward artificial general intelligence and its successors until we can pursue it responsibly. That means investing seriously in alignment research, not just control techniques. It means establishing international frameworks that prevent a race to the bottom between companies and countries. And it means being honest about what we don’t know, rather than betting the future of humanity on the assumption that things will probably work out because they have before.

The window for shaping how this technology develops is still open. But it won’t stay open forever. If you’d like to understand why I believe AI is the defining issue of our time, check out my previous posts. And if you want to learn more about AI, how it might affect our lives and what you can do about it, please visit my recent passion project at feeltheasi.com.

1. GLP-1 receptor agonists trace back to a researcher studying Gila monster venom for diabetes, where weight loss was a complete surprise. Darwin’s theory of natural selection crystallized only after he read Thomas Malthus’s essay on population growth outstripping food supply. The asteroid impact theory for dinosaur extinction came when a physicist applied nuclear trace analysis to a geological clay layer that paleontologists had been studying for years without explanation. Google’s PageRank algorithm was born when Larry Page recognized that web hyperlinks work exactly like academic citations, an insight from 1950s library science that no computer scientist had thought to apply. Dan Shechtman won the Nobel Prize for discovering “impossible” crystals whose structure had already been predicted by a mathematical physicist’s aperiodic tiling patterns that the materials science community had largely overlooked. Daniel Kahneman, a psychologist who never took an economics course, won the Nobel Prize in Economics by applying known cognitive bias research to challenge the foundational assumptions of economic theory.

2. Back-of-napkin math: current frontier AI models output 50-100+ tokens per second, roughly 10-20 times faster than a human reads (~250 words per minute). Training a frontier model requires far more compute than running it for inference, so the same cluster could comfortably run tens of thousands of copies. At 10,000 copies each operating 10x faster than a human, you get 100,000 days of work by ASI for every day that passes in the real world, or roughly 274 years. Leopold Aschenbrenner’s Situational Awareness projects that near-future GPU fleets could support millions of simultaneous copies, yielding what he estimates at “100 million human-researcher-equivalents.” The AI 2027 research project models 200,000 copies running at 30x human speed on a 200,000-GPU cluster. Advances in model compression (e.g., Google’s TurboQuant) could push these numbers higher still. My estimate of ten thousand copies at 10x speed is a conservative one.