In humanity’s quest to constrain the dangers from fast-advancing artificial intelligence, many have looked to the most revolutionary technology that preceded it: nuclear weapons. SpaceX CEO Elon Musk has called artificial intelligence “far more dangerous than nukes.” Anthropic CEO Dario Amodei, who assigns The Making of the Atomic Bomb to employees and has likened himself to the Manhattan Project scientist Leo Szilard, describes the most powerful AI models as “weaponizable nuclear materials.” When asked how a technology such as AI can be regulated, OpenAI CEO Sam Altman often references the world’s nuclear materials watchdog, the International Atomic Energy Agency. Even the director of the CIA, John Ratcliffe, has made the same comparison.
These comparisons capture real parallels. Like nuclear weapons, frontier AI is powerful and hard to control. Like nuclear technology, it can be used for good or for ill. And like the Cold War arms race, the race for AI breakthroughs may reorder the global balance of power. The nuclear analogy, then, seems to offer a blueprint for how to effectively govern a promising but potentially dangerous technology such as AI.
But even if efforts to control the bomb may be a source of inspiration, they are not a ready-made template for AI governance. The analogy paints too rosy a picture of nuclear weapons governance, crediting it with successes that are still debated and may yet be unearned. It also blurs key differences between AI and nuclear technology that reveal just how much more difficult AI governance will be. Nuclear materials can be traced and accounted for to a much greater extent than AI components can. Nuclear weapons were designed, built, and now remain under strict government control; AI is built principally by private companies with strong profit motives, which move faster than states can regulate them. Whereas nuclear technology’s use cases are narrow, AI’s cut across nearly every sector and security domain.
AI governance also faces an additional challenge: policymakers need to mobilize support for action even as the scale of the threat remains abstract. Recent disclosures by OpenAI, Anthropic, Google, and Meta that their advanced models escaped testing environments and gained unauthorized access to the Internet have raised alarm in Washington about the risks of frontier AI. But the implications of these incidents remain ambiguous. They are removed from most people’s everyday experience and have not yet caused large-scale harm, so they are unlikely on their own to generate the sustained political pressure needed to build and maintain a durable oversight regime. Despite the many warnings from researchers and policymakers about AI’s myriad risks, the technology has yet to experience a decisive moment, such as when nuclear weapons were used at the end of World War II, which transforms fear into action. But the world cannot wait for such a moment to act. Looking to history may be a productive place to start—but it cannot provide a ready-made blueprint for the problem that AI poses. Artificial intelligence is unlike any technology of the past, and the order that governs it must be, as well.
IMPROBABLY GOOD LUCK
The most ambitious international visions for regulating AI borrow from nuclear governance in the belief that such a framework can both help minimize AI’s potential harms and boost the United States’ ability to lead. Haydn Belfield, a research scientist at Google DeepMind, has proposed an arrangement modeled on the Nuclear Nonproliferation Treaty (NPT) that would allow only certain states with sufficient domestic regulation to possess powerful chips—a nonproliferation regime for AI. Others, including Altman, argue for an independent body of experts with broad inspection and verification powers over AI, more akin to the IAEA. A third cluster seeks to re-create the arms control practices and mutual vulnerability of the Cold War to stabilize the AI race between the United States and China.
The allure of these precedents is understandable. The nuclear order is among the few successes of postwar global governance. In a 1953 speech to the United Nations, then U.S. President Dwight Eisenhower outlined a plan, which became known as “Atoms for Peace,” aimed at curbing the technology’s risks while promoting its benefits: countries could access nuclear technology so long as they promised to forgo using it for weapons.
The proliferation problem wasn’t solved immediately. In 1963, President John F. Kennedy predicted that 15 to 25 states would become nuclear powers in the following decade, and many senior U.S. leaders believed a nuclear war was inevitable. But by the 1970s, the nuclear order’s cornerstone treaty, the NPT, entered into force, and the IAEA’s expert-led inspections and verification regime, backed by Washington and Moscow, had begun to police it. The two superpowers negotiated test bans, convened arms control summits, and set up a crisis hotline. Today, the NPT commands virtually universal international assent. Only nine states possess nuclear weapons, and no nuclear war has occurred. Many take this to mean that humanity found the correct balance between preventing harm and promoting peaceful use. And if that balance could be found for nuclear weapons, perhaps it can work for AI, too.
The actual history of nonproliferation and arms control shows something more muddled: in many cases, it was superpower coercion and cajoling—and sometimes sheer cost—that thwarted nuclear aspirations. South Korea and Taiwan gave up nascent weapons programs after intense pressure from the United States, backed up by alliance guarantees, market access, and a threat to pull military support if they did not comply. Israeli airstrikes took out the Iraqi and Syrian programs; Libya surrendered its own under sustained U.S. and British pressure. Others gave up the bomb for internal reasons. Sweden and Switzerland did so because of its high costs, and Argentina and Brazil did when democratic transitions redirected their foreign policies toward regional cooperation. South Africa built six warheads before dismantling them as the end of apartheid transformed the country’s domestic and foreign priorities.
That no frontier AI model has yet caused a catastrophe may well be luck.
When states were determined to develop nuclear weapons and could withstand outside pressure, the nonproliferation regime had little power to stop them. India, Israel, North Korea, and Pakistan all managed to build the bomb. The order has thus always been incomplete—reliant on the collusion of nuclear states and their willingness to resort to force if need be to stop proliferation, and often unable to dissuade those determined to obtain nuclear arms. Even existing nuclear powers do not always cooperate; China and France, two of the five nuclear weapons states in the NPT, didn’t join it until 1992.
The nuclear nonuse record is just as contingent as that of nonproliferation. Conventional wisdom credits a suite of bilateral tools for the 81-year-long nuclear peace: arms control talks, hotlines, notifications and warnings, and calibrated deterrence postures kept the competition between Washington and Moscow from escalating past the point of no return. But although these interventions speak to the broad success of the nuclear order, a closer look shows how often catastrophe was averted only at the margin.
During the Cuban missile crisis, in October 1962, Washington and Moscow came within a decision or two of a nuclear war. As Sergey Radchenko and Vladislav Zubok argued in Foreign Affairs in 2023, Soviet Premier Nikita Khrushchev’s mission to station nuclear weapons in Cuba was a poorly planned gambit that required improbably good luck from the start. Many close calls during that period could have easily precipitated nuclear use. U.S. plans to invade Cuba were developed without knowledge of the extent of the Soviet nuclear presence; a Soviet submarine came close to launching a nuclear torpedo at U.S. naval forces in the Caribbean, and after a U-2 spy plane drifted into Soviet airspace, it had to be escorted home by nuclear-armed U.S. interceptor aircraft to protect it from a Soviet pursuit. Many close calls during the crisis occurred far down the chain of command, without Kennedy’s or Khrushchev’s direct knowledge or involvement. Recent analyses show how the respective leaders’ growing sense that they could not control events, rather than their desire to defeat the other, helped end the crisis and spurred subsequent efforts to negotiate superpower restraint.
A series of smaller nuclear-related incidents followed the Cuban missile crisis, including false alarms, misread exercises, and nuclear bomber crashes. The increasing reliance of U.S. nuclear posture on intercontinental and sea-based missiles, which can strike adversaries far quicker than bombers can, added to these incidents’ danger by giving decision-makers less time to identify and correct mistakes. New warning systems, designed to detect such attacks, were not infallible, either. In November 1979, a U.S. live warning system mistook a test tape for a major Soviet attack; before it was confirmed to be an error, at least ten fighter aircraft were dispatched and the president’s doomsday plane took off without him on it. In September 1983, Soviet Lieutenant Colonel Stanislav Petrov’s early warning system reported an incoming U.S. strike, but he correctly judged it to be a malfunction.
In these cases, the institutions built around nuclear nonproliferation and nonuse did not preclude possible catastrophe. As more archives on nuclear history have opened to researchers, more near misses have come to light. These discoveries open the question of how much of the nuclear order’s success is owed not to design but to luck. Crediting the institutions while forgetting chance will leave architects for AI governance ill-prepared for what is likely to come. That no frontier AI model has yet caused a catastrophe may well be luck. And luck can run out.
TESTING THE LIMITS
The nuclear example is even less helpful when considering how tenuous weapons control arrangements have become. Relations between Russia and the United States, which together still possess the vast majority of nuclear weapons, are at a post–Cold War nadir. The main treaties governing their nuclear arsenals—New START for strategic forces and the Intermediate-Range Nuclear Forces Treaty for regional ones—have died. Russia’s war in Ukraine and European concerns about U.S. commitments to NATO have prompted many of Washington’s allies to discuss other nuclear-sharing options, and officials in some member states, such as Poland, have openly discussed acquiring the bomb.
China continues to expand its arsenal unconstrained by any treaty or external oversight: its stockpile has nearly tripled since 2020, and the Pentagon projects that Beijing will reach 1,000 warheads by 2030. Washington’s Pacific allies are also uneasy. In South Korea, polls consistently show majority support for a domestic arsenal to counter North Korea. In Japan, which has some of the world’s strongest antinuclear sentiment, politicians are increasingly testing the taboo, openly debating nuclear deterrence as part of the country’s defense.
To some, the treaty at the heart of the nuclear order, the NPT, remains the last obstacle between nuclear order and anarchy. But even the treaty’s future seems uncertain. As part of its core bargain, the NPT calls on its nuclear “haves” to work in good faith toward nuclear disarmament, a central reason the “have-nots” saw fit to take the deal and give up nuclear weapons in perpetuity. But trends in the last decade point toward growing and modernizing nuclear arsenals, not shrinking ones. Impatient with unfulfilled promises, more than 90 states—roughly half the NPT’s nonnuclear membership—have signed a rival treaty that bans the possession of nuclear weapons outright. Nuclear weapons states and their allies have stated they have no interest in signing such a treaty and accuse it of undermining the existing order.
The limits of the nuclear nonproliferation order are on full display in the Middle East, where every effort to peacefully control nuclear technology has failed. Although Israel has never admitted possessing the bomb nor joined the NPT, it weaponized its program in the 1960s and has attacked its neighbors’ nuclear industries to maintain its regional nuclear monopoly, destroying unfinished Iraqi and Syrian nuclear reactors in 1981 and 2007, respectively. International efforts to prevent Iraq from proliferating culminated in the United States’ 2003 invasion. The United States is now at war in the region once again, this time with Iran, which it has justified in part with the dangers posed by Tehran’s nuclear program.
NOT LIKE THE OTHER
There are reasons more fundamental than a mixed historical record that the nuclear blueprint is not perfectly suited for AI. The nuclear order’s machinery is effective because the materials needed to build nuclear weapons are traceable, measurable, and physically constrained. To develop nuclear weapons, a state needs meaningful quantities of specific isotopes of just two elements, uranium and plutonium. Both are expensive to produce, can be measured and tracked, and rely on large, complex facilities for production and processing. Plutonium, moreover, does not occur in nature beyond trace amounts. The relatively small amount of extra work required to enrich reactor-grade uranium to weapons-grade uranium means that inspectors must be vigilant in case states seek to sprint to a bomb. But it is still possible for outside observers to sketch a rough picture of what a state can do with the nuclear technology it has and to determine whether its capabilities are tipping toward weaponization.
The buildout of AI systems leaves physical artifacts, too. It requires advanced chips, semiconductor manufacturing equipment, photolithography machines, fabrication plants, and, above all, data centers, which makes the process observable and partially traceable. The current AI expansion is among the largest industrial mobilizations in history: training clusters, or networks that link thousands of advanced AI chips inside data centers, occupy hundreds of acres, draw as much power as a midsize city, and are visible from orbit. Most proposals around AI governance wisely focus on this physical chain because concentrated high-performance compute is, for now, the most tractable lever constraining new actors’ ability to build advanced AI systems.
But AI’s physical signature differs from that of the bomb in ways that would frustrate a nuclear-style approach to identifying when actors have reached a certain threshold of capability or when they pose a threat. Once a state has facilities to produce fissile material, observers can estimate how much weapons-grade material it has and can therefore roughly estimate how many bombs it can produce. The relationship between AI infrastructure and capabilities, however, is not fixed. Compute is strongly correlated with capability; over the last decade, more compute and more data have yielded more powerful systems. But the mapping is loose and shifts over time. Because algorithmic efficiency improves relentlessly, what requires cutting-edge resources one year becomes far cheaper and easier the next, and large models’ abilities can be distilled into much smaller ones.
Models might contribute to calamitous scenarios without directly causing them.
Even when training ends, AI systems can still improve. The same system can produce different outcomes depending on what software surrounds it and what tools it can access. Models can improve, for example, when they are simply given more time and computing power to work through problems. Sophisticated modern nuclear weapon designs can stretch a given amount of fissile material further, but the total stockpile of materials limits the size of a state’s nuclear arsenal and capabilities. The compute needed for AI training does not impose a comparable constraint. China’s ability to keep up with American AI development despite its shortage of compute best demonstrates this fact.
AI also has a “foundry problem”: once a model is trained, its weights become separable from the infrastructure that created it. Model weights—the numerical parameters that define a model after training—can be copied, transmitted, modified, and deployed far from their source, and developers, most prominently but not exclusively in China, make them public to encourage adoption. These open-weight models can be downloaded and modified by anyone, and they are now close to the U.S. frontier on many measures, trailing the most powerful models by months rather than years. The world of nuclear weapons holds no comparable analog to the public release of a generally usable, improvable, and potentially transformative capability.
Deploying a frontier model at an economically or militarily meaningful scale takes significant amounts of computing power. Some governments have imposed chip export controls in the view that restricting chip access can prevent an adversary from amassing enough computing power to train and run frontier models. Still, any leverage in compute is partial—and weakening. Stolen or published weights can be run almost anywhere, and the minimum capacity needed to produce useful results keeps falling. A single model can serve purposes its builder never intended: reading medical records, finding software vulnerabilities, reasoning about weapons design. Although supply chain controls can slow frontier development, raise costs, and make developers’ activities easier for regulators to observe, they cannot readily answer the question a nonproliferation regime exists to answer: what actors are using which capabilities, and to what end.
This problem would confront any treaty on AI. A hypothetical U.S.-Chinese agreement on AI development and testing standards would almost certainly require a credible means of compliance verification to be politically viable in either country, which would depend on common definitions of what, exactly, should be tracked. In nuclear arms control, the objects of verification are well defined: missiles, launchers, bombers, and warheads. These efforts and agreements, moreover, took years to mature. Complete verification of compliance with an AI agreement might involve transparency measures that states would find too intrusive. Governments engaged in nuclear arms control know exactly what the enterprises are up to and can therefore answer for them. That is not the case for AI labs.
Finally, frontier AI and nuclear weapons differ in terms of who controls the technology. The bomb was developed under government control and oversight. Building and keeping a nuclear arsenal requires state-level coordination and durable state institutions, including a well-organized military and a nuclear scientific complex. Every government that has built a nuclear energy program or sought nuclear weapons has worked to strengthen those state institutions. AI is the opposite. It was created by independent actors—researchers, scientists, founders, and commercial developers—whose speed has so far outpaced governments’ efforts to regulate them.
CATASTROPHE IN THE ABSTRACT
The final difference between nuclear weapons and artificial intelligence may be the most consequential. Nuclear weapons were first introduced to the world in a violent display: the bombings of Hiroshima and Nagasaki, in August 1945. The nuclear order developed from a concrete rather than speculative demonstration of the technology’s catastrophic potential to cause harm. Hundreds of subsequent nuclear tests showcased the technology’s ever-growing destructive potential. Whatever else people contested about nuclear weapons, no one disputed the horror that would result from their use.
AI, on the other hand, did not emerge from an equivalent event, nor has it yet produced one. This summer’s Hugging Face incident—in which advanced models escaped software constraints and breached external systems, including the AI company Hugging Face—spurred a new round of warnings about AI control and oversight across the AI community, including from developers themselves. But its consequences have so far remained limited, and the debate over its meaning for AI governance largely confined to specialists. AI’s truly catastrophic scenarios—engineered pathogens, cascading infrastructure failure, the loss of meaningful human control over misaligned superintelligent systems, and even human extinction—remain abstract and publicly contested.
Absent a catalyst that clearly demonstrates AI’s potential for devastating harm, policymakers who seek to govern the technology must do so without the political energy and capital that built the nuclear order. The AI harms that generate the most political attention—the effects of chatbots on young people, job displacement, the local costs of data centers—have led to legislation, litigation, and political backlash. But they operate in a different register from the catastrophic risks that motivate the nuclear comparison. Even for nuclear weapons, the fading memories of Hiroshima and Nagasaki and the passing of the final survivors of those events could foment a renewed complacency.
Even if an AI-driven catastrophe were to occur—as many at the frontier labs now warn will happen without a change in trajectory—it might not concentrate public and elite concern on the right policy levers. The Cuban missile crisis sparked Cold War arms control efforts because both Washington and Moscow clearly understood they had come unacceptably close to the precipice of Armageddon. A misaligned model that bypasses safeguards and causes a major cyber-incident resulting in a loss of life or wealth would be hard to mistake for anything else. But models might contribute to many other calamitous scenarios without directly causing them. A bioweapon, a market collapse, or an energy blackout might be connected to AI misuse, but the extent to which AI determined the outcome could be difficult to establish with any certainty. Messy causal chains would allow a variety of stakeholders to shift blame, diffusing the urgency of public calls for stronger regulations that could prevent a recurrence.
AT THE MARGIN
If emulating the nuclear order’s focus on controlling material inputs is inadequate for AI, then efforts to build an AI governance regime must also examine the process of how models are designed and how they are applied. Tracking what kind of tests a developer runs before deployment, what they show, and who besides the developer is allowed to audit the model’s training and outputs may make it easier to reduce some of AI’s risks. So will focusing regulations on AI’s applications, alongside its models and infrastructure. Decisions about the safety of AI models cannot be made with regard to general-purpose systems; controls must be imposed on particular domains where the consequences of misuse are high. Some priorities are already clear: most frontier developers agree that AI capabilities in synthetic biology and offensive cyber-operations warrant particularly strong safeguards.
The problem with constructing a durable governance framework, as in the Cold War, is more political than technical. In the United States, the Trump administration has resisted new AI regulation and oversight, arguing that such measures would impede innovation and advantage China. Earlier this month, U.S. President Donald Trump wrote on social media that “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT.” Yet this deregulatory stance has coexisted with spurts of case-by-case executive intervention. In June, the White House used export controls to cut off foreign access to Anthropic’s Fable 5 and Mythos 5 models, before restoring it at the end of the month, and pressed OpenAI to limit its release of GPT-5.6 to government-approved actors. Such an uneven federal approach to AI regulation serves neither developers, who would benefit from predictable rules to plan investments and releases, nor the public, which needs safeguards rather than ad hoc intervention.
The federal government has plenty of instruments that it could use to more durably regulate AI. In addition to export controls, it can invoke the Defense Production Act to compel firms to disclose sensitive information and use federal purchasing power to condition what agencies buy on what developers share and submit to outside evaluation. What Washington lacks is a durable framework to guide these measures and the determination to enact them. The tools now in use—export licensing, informal pressure on individual firms—are improvised and are wielded largely at executive discretion, meaning they can be reversed by whoever occupies the White House next. The political wrangling in the United States after World War II over whether nuclear weapons should be controlled by a military or civilian authority—which ended with the military custody of the weapons, after a brief period of civilian control—produced what AI governance still lacks: clear statutes, defined federal oversight, and settled answers on domestic governance.
The federal government has plenty of instruments to regulate AI.
Progress along the United States’ other regulatory pathways has also been piecemeal. Lawmakers in Washington have introduced federal bills addressing parts of the risk landscape—such as age restrictions on chatbots and a kill switch for dangerous models—but these face a long road through Congress. Efforts by state governments in California, Illinois, and New York to govern frontier AI firms through mandatory incident reporting, transparency, and auditing provisions have closed some of this gap, but the laws won’t become binding until 2027 or 2028 and are limited by the states’ narrower jurisdictions and industry-friendly carve-outs. No bill has yet proposed to mandate oversight regimes with the authority and breadth that regulate other safety-critical industries such as aviation and nuclear energy. Congress will need to build this kind of machinery to give policymakers a shot at effectively regulating AI.
If policymakers are to succeed in designing a governance regime for AI, they must both draw the right lessons from the past and innovate in the dark. Those who built the bomb were forced to imagine a framework for an “absolute weapon” for which they had little precedent. They began by reckoning with the bomb as it actually was, often motivated by fear of its consequences. By looking back at the nuclear order’s treaties, organizations, and mechanisms, AI’s titans are inverting this process, starting with the institutions and then surmising how to make the technology fit them. This has obscured those institutions’ failings and fragility—and what makes AI so unique and dangerous.
The coming summit between Trump and Chinese leader Xi Jinping this month will be an important test for whether the two leading AI powers can find any ground for cooperation on AI. AI safety is high on the agenda of potential items to discuss. But whatever comes of the summit, the more urgent and consequential work lies at home. Washington does not even yet know exactly what U.S. firms will be capable of in the coming months. Seeking international control without settled domestic oversight would answer the right questions in the wrong order. Until Congress provides clearer answers about who oversees the AI frontier, what firms must disclose, and what powers regulators possess, American approaches to managing AI will remain diffuse and improvisational. In the most meaningful parallel between AI and the bomb, there is little margin for error.
Loading…
