The Plausible Techno-Dystopian Future

AI apocalypse fears went mainstream last month after an Anthropic researcher resigned, saying the technology has more than a 10% chance of wiping out humanity. Leaders of AI companies called for a collective slowdown of development. 

In the process, the public learned that the tech industry’s conception of AI safety is dominated by a small group of people who believe, with a certainty and devotion resembling religion, that an artificial superintelligence will arrive soon and take over the world. Befitting that millenarian outlook, they appear to have an unhealthy internal culture, part of which sure looks like a sex cult. 

But while existential risk deserves some attention, fixating on apocalyptic scenarios ignores real-world problems involving the actually already existing AI. We have proven methods to address disruptive technology, but for anyone who opposes the use of those methods, apocalyptic fantasies can be a useful distraction.

The problems arising from AI development are widespread and multiplying—and they often come from people overrating the technology’s capabilities, not underrating them. A terrible example is how the U.S. military over-relied on AI for targeting in Iran and as a result bombed an elementary school, killing over 100 children. We have disturbing testimony from Israeli military officers that AI target selection made them feel morally absolved when they killed civilians, since they felt that it was the machine which made the choice. In a similar, nonmilitary example of ‘outsourcing’ a harmful, morally weighted decision, UnitedHealth allegedly used an AI model with a massive error rate to conveniently and profitably deny insurance claims.

Other AI problems can and do arise not from its incapacities and errors but from the models providing useful but harmful information. AI chatbots have encouraged people to commit suicide—and taught them how to do it. ChatGPT helped a school shooter plan an attack that killed eight and injured 27. AI also assisted a shooter who killed two at Florida State University last year, recommending a time, location, and weapon to maximize deaths.

On top of life-and-death concerns, AI slop is flooding the internet and accelerating the spread of misinformation. It’s making propaganda better designed and cheaper to produce. It’s increased cheating in education, and it’s causing learning loss. It’s disrupting the job market in ways we’re only starting to understand. And a lot more.

But these concerns are barely part of the AI safety conversation, because the national discussion is fixated on supposed existential risks, and on the tech industry’s preferred approach to them. “We all need to hope it’s an engineering problem,” NVIDIA CEO Jensen Huang recently argued when asked about AI somehow destroying the world. Because “if it’s not an engineering problem, it’s not solvable.”

Well, maybe not by engineers, especially those who think today’s main problem is an imagined future dystopia. But large language models (LLMs) are far from the first innovation to change society, and there’s precedent for addressing society-changing innovations with government policy, rather than putting our faith entirely in industry leaders.

Jamelle Bouie compares AI to railroads, an incredibly disruptive innovation that transformed society and the economy, while concentrating wealth—and by extension political power—in the hands of a few owners. Faine Greenwood compares AI to drones, a relatively new innovation that has been used to kill people in the real world, and increasingly so through the arms race between Russia and Ukraine. David Robinson, an OpenAI employee who resigned over safety concerns, compares AI to nuclear power plants and busy airports, both of which build in redundancies to avoid catastrophe. 

Another possible comparison for AI is nuclear weapons, which ushered humanity into a new age and really can kill everyone on Earth, right now. The world has lived under the shadow of apocalypse since 1945. We survived the Cold War not primarily because of engineering, but because of the institutions we built to manage that dread risk.

None of these analogies could be exact. Railroads transformed the physical landscape while AI is primarily transforming the digital one. Drones are real-world weapons and surveillance platforms, while AI could only direct or take input from platforms. Power plants and airports are physically contained, while AI is software and can access other systems it connects to. Nuclear weapons take large, industrial processes to make, and a primary non-proliferation focus is regulating radioactive uranium, while AI is more widely distributed, more controlled by the private sector, and has different types of risks.

We’ve dealt with society-altering technological change before—channeling it to elevate rather than harm the public interest. We’ll have to work out new specifics for AI, but we have a proven approach to use, and it does not rely on putting all our trust in the industry and its evangelists under the assumption that they’re the only people who can understand what’s happening and know what’s best for everyone.

One of the most discussed hypothetical risks is AI designing new viruses, but around 70% of all “AI can make bioweapons” literature comes from or is funded by the same small group of people referenced earlier who are unhealthily fixated on apocalyptic hypotheticals. However, as Martin Pfeiffer explains, bioweapons are quite hard to make, and human beings will be centrally involved in biolab processes for the indefinite future. LLMs can advance bioscience by finding patterns in massive amounts of information, such as DNA sequences, but that’s a far cry from designing new pathogens and actually building one, let alone doing so without human involvement. 

We already know that research on pathogens is dangerous and we regulate it extensively. There’s no good reason to think that tight safety standards, monitoring, and enforcement will be unable to handle biolabs that utilize new, more advanced software. Biosecurity also provides a model for the distributed threat of AI. Since gene-mapping and -modification advancements in the late 20th century, biosecurity has worried that a single lab anywhere could produce a novel pathogen, deliberately or accidentally.

Similarly, cybersecurity provides a framework for addressing many AI risks. Cyberweapons such as Stuxnet (2007–2010) and NotPetya (2017) were complex malware that autonomously spread, and ended up acting in unintended ways. Reports about OpenAI’s systems “going rogue” and unexpectedly accessing government websites look different in that light. The technology is more advanced, but the problem isn’t new.

Outside of Silicon Valley, doomerist paranoia about AI holds less sway. China is mostly optimistic, dismissing apocalyptic fears as Western, perhaps an attempt to stifle Chinese development. Arthur Mensch, CEO of French AI company Mistral, recently shrugged off fears of rogue agents, blaming companies for carelessness and arguing “AI is software, it can be controlled.”

It can at the societal level too. AI is transformative technology, and the way to handle that is government regulation, legal liability, antitrust enforcement, and international treaties. These tools have worked many times—railroads, oil, telecommunications, air traffic control, nuclear weapons, biotechnology, and more—harnessing the benefits of new innovations while mitigating the downsides.

One tool that should be applied as soon as possible is holding AI companies liable, civilly and in some cases criminally, for the actions of models they build and deploy. That would, at minimum, disincentivize the sort of recklessness seen in the Hugging Face incident, when OpenAI agents escaped their “sandbox,” accessed the internet, and breached Hugging Face’s systems. Unauthorized computer access (i.e., hacking) is already illegal—though arguably not if it’s accidental—and OpenAI’s own assessment says it had lowered security barriers for the test. Companies are already liable under civil law when their negligence or recklessness causes harm, with lawsuits forcing businesses to pay large sums to compensate for tobacco-related health problems, opioid addiction, oil spills, and more. 

For an example of the alternative viewpoint, OpenAI CEO Sam Altman recently argued that “the world should accept some bad things happening for the benefit of this technology.” That might make sense for a person getting incredibly rich and powerful off the technology while worrying about competition from other companies, including foreign ones. But the rest of us don’t have to agree that these people and their work are so special that, for the first time ever, law and public interest don’t apply.

In the short-term, we need a change in government. The Trump White House policy is to trust industry leaders to self-regulate, and the president’s only apparent priority is getting people to say “superintelligence (SI)” instead of “artificial intelligence (AI).” But with sizable majorities of Americans disapproving of AI and the data centers necessary to train and run AI models, that change in government will likely come. In the meantime, the states and the EU should take the lead.

Beyond that, the main obstacle we need to get past is the tech oligarchs, the cultish doomsayers in AI safety, and the public deference they have cultivated through grandiose promises and apocalyptic fears. The plausible techno-dystopian future we’re staring down is not one where a digital demon kills us all, but one where we recklessly develop AI and shove it into everything, absolve ourselves of responsibility for decisions ‘made by AI,’ and concentrate immense wealth and control of information in a small number of tech-industry figures whose personal interests do not align with the public interest.

Featured Image is The great race for the Western stakes 1870, by Currier & Ives.