From copyright to catastrophe: what gets lost when the AI debate turns existential?

AI extinction risk is back in the headlines. But while the AI safety debate turns to superintelligence, kill switches and whether humanity survives, much more immediate questions about copyright, jobs, personal data, energy, democratic control and who pays for the AI transition are being decided now.

Photo: Ron Lach / Pixels

Could artificial intelligence really wipe out humanity by the end of the decade?

It is a terrifying and important question, but it is not the only question we should be asking.

The AI debate is increasingly being pulled between two enormous stories about the future. In one, artificial intelligence transforms human life for the better: curing disease, accelerating science, increasing productivity and creating abundance. In the other, increasingly capable systems escape meaningful human control and, at the extreme, threaten humanity itself.

Those stories look like opposites. But they have something important in common: both direct our attention towards a future that has not happened yet.

Meanwhile, much more immediate decisions are being made about who owns the data used to build AI, whose work it may replace, who captures the economic gains, what happens to wages and bargaining power, what resources AI infrastructure consumes, and how much control governments and the public retain over systems increasingly embedded in everyday life.

Catastrophic risk deserves to be taken seriously. But we also need to ask what slips out of view while our attention is fixed on either the miracle or the apocalypse.

Why is everyone talking about extinction again?

Over the past week, the apocalypse has become very difficult to ignore.

This month, UK viewers were invited to become “apocaloptimists”. The AI Doc: Or How I Became an Apocaloptimist, directed by Daniel Roher and Charlie Tyrell, arrived on Netflix UK after premiering at Sheffield DocFest earlier this year. It was deliberately made as an accessible introduction to artificial intelligence, exploring both its extraordinary possibilities and its potentially existential dangers.

At almost exactly the same time, Jacob Coxon, a British researcher who had worked at OpenAI and Anthropic, resigned from Anthropic with an extraordinary warning. He accused frontier AI companies of racing towards self-improving superintelligence while “gambling with our lives”, saying that people building the technology genuinely believe it could kill us all by the end of the decade.

He was not alone. Anthropic CEO Dario Amodei followed with an essay, We Must Pace the Frontier, arguing that capability was advancing “drastically faster” and that frontier development needed to slow long enough for safety research and independent evaluation to catch up. Sam Altman and Elon Musk backed parts of his call for restraint, while President Donald Trump emphasised the importance of maintaining the United States’ lead over China and argued against a significant regulatory slowdown.

Then, on 17 September, King Charles convened executives from Nvidia, Google DeepMind, OpenAI and Anthropic, alongside government and civil-society figures, at Dumfries House in Scotland. The King spoke openly about both AI’s potential and its “existential dangers”, asking how human beings can ensure that the technology remains under meaningful control.

By Friday morning, the Guardian was inviting readers to a live Q&A under the rather direct headline: “Could AI really end humanity?”

That is quite a lot of apocalypse for one week.

We will come back to what this debate may be pushing aside. First, though, the catastrophic argument deserves to be taken seriously on its own terms.

Haven’t we been here before?

The current fear around AI feels new, but it isn’t.

In March 2023, the Future of Life Institute published an open letter calling for at least a six-month pause in training systems more powerful than GPT-4. It was eventually signed by more than 30,000 people, including Elon Musk, Yoshua Bengio and Stuart Russell, and warned of an “out-of-control race” to build increasingly powerful systems that even their creators could not reliably understand, predict or control.

No coordinated pause followed.

Two months later, the Center for AI Safety published an even shorter statement: mitigating the risk of extinction from AI, it said, should be treated as a global priority alongside pandemics and nuclear war. Signatories included Geoffrey Hinton, Yoshua Bengio, Sam Altman, Dario Amodei and Google DeepMind’s Demis Hassabis.

By November 2023, representatives of 28 countries, including the United States and China, had gathered at Bletchley Park for the first global AI Safety Summit. The resulting declaration acknowledged the possibility of serious or catastrophic harm and committed governments to international cooperation. Importantly, it did not commit them to stopping development.

If everyone is frightened, why not just slow down?

This distinction is important because AI is not developing in a political vacuum.

The United States and China increasingly treat advanced AI as a strategic capability. If both believe increasingly powerful AI might be dangerous, cooperation makes sense. If both also believe it may determine future economic, technological and military power, being the one that slows down first looks less attractive.

China signed the Bletchley Declaration, but it did not agree to a moratorium. After Amodei’s latest call to pace frontier development, China’s state-backed Global Times characterised parts of his proposal, particularly tighter controls on advanced chips and Chinese access to US technology, as a “Cold War” approach designed to preserve American dominance. US political debate, meanwhile, repeatedly returns to the fear that stronger domestic restrictions could allow China to overtake it.

There is a slightly horrible logic to this: everybody may have an interest in restraint while nobody wants to restrain themselves first.

That tension became even more visible on Saturday, when it was reported that Anthropic was considering another model release in response to competitive pressure from OpenAI. The report neatly captures the practical problem at the heart of Amodei’s argument: unilateral restraint is difficult when rivals keep moving.

The Cold War comparison is useful, but in one respect it may be too comforting. Nuclear weapons are terrifying, but they are comprehensible in a way that advanced AI is not. Most people broadly understand what a nuclear weapon is, what it is designed to do and what its use could mean. We have seen Hiroshima and Nagasaki. We understand radiation, blast zones and mutually assured destruction. The devastation is horrifying, but the nature of the threat is understandable. AI is very different.

Even highly knowledgeable people disagree about what increasingly capable systems will ultimately be able to do, how quickly those capabilities will develop, and what their long-term effects on society might be. We do not have a shared picture of what an AI catastrophe would actually look like. There may be no mushroom cloud, launch warning, country-wide alerts on your mobile phone, or a single moment at which everybody realises that something has gone very badly wrong.

More unsettling still, even the companies building the most advanced models cannot fully explain everything happening inside them.

Researchers understand the architectures, mathematics and training processes very well. What remains difficult is interpreting the internal representations and strategies that emerge during training. Anthropic has described those learned strategies as arriving “inscrutable” even to the model’s developers, acknowledging that researchers still do not understand how models accomplish much of what they do. 

We are contemplating the possibility that these systems might become extraordinarily powerful while simultaneously developing an entire field of research devoted to understanding what is happening inside systems we have already built. 

Unlike a nuclear weapon, AI does not sit visibly inside a missile silo waiting to be deployed. It is distributed through data centres, cloud infrastructure, software, phones, search engines and workplaces. It increasingly sits inside the systems through which we communicate, make decisions and encounter information about the world. If the most catastrophic warnings about advanced AI are even partly correct, the danger may be extremely difficult for the public to perceive and understand. The technology can become more capable, more autonomous and more deeply embedded in ordinary life without looking remotely apocalyptic. 

This makes the competitive logic particularly uncomfortable.

Governments may simultaneously believe that increasingly powerful AI requires greater control and that national security requires them to build it faster than their rivals. The technology becomes more powerful because nobody wants to be the country that falls behind. And the more deeply it becomes embedded in economic and informational infrastructure, the harder the idea of “turning it off” begins to look.

Perhaps the most unsettling thing about all of this is also the simplest: we do not know.

So, could Jacob Coxon actually be right?

Possibly. But “AI could kill everybody by 2030” is not an established scientific forecast. It is the end point of a chain of assumptions, and every link in that chain is contested.

For the most catastrophic version of the argument to work, AI capabilities would need to continue advancing extremely rapidly. AI would need to become increasingly capable of contributing to its own development. More autonomous systems would need to acquire enough agency, access or capability to behave in ways their operators could not reliably predict or control. Existing mechanisms for containment, shutdown and human oversight would then need to prove inadequate. And failures at that level would somehow have to translate into harm on a genuinely catastrophic scale.

Every link in that chain is contested, but each is plausible enough to deserve scrutiny. So, let’s take the chain apart.

Is AI still getting better fast enough?

The first assumption is that capability continues to advance very rapidly. The real concern is what happens if AI becomes increasingly good at building better AI.

This is where the idea of the technological “Singularity” enters the argument. The term is usually used to describe a hypothetical point at which technological progress becomes so rapid, potentially because increasingly capable AI helps to develop ever more capable AI, that what follows becomes extraordinarily difficult for humans to predict. 

We are nowhere near demonstrating that such a point is inevitable. But one possible route towards it would be a shortening feedback loop between AI capability and AI development.

Anthropic is already measuring this inside its own research. As of August 2026, it says Claude “leads” 26% of its measured AI R&D work, completing most of the task from a high-level prompt while a human supervises, and collaborates on more than 90%. Crucially, Anthropic also says Claude is not operating fully autonomously on any measured subset of that work.

This remains far short of recursive self-improvement in the science-fiction sense. Humans still choose goals, provide infrastructure, decide what gets trained and evaluate outcomes. But the loop is becoming shorter: AI is increasingly doing work used to build the next generation of AI.

How autonomous are these systems really?

AI systems are not independent organisms roaming around the internet. They run on physical computers and need electricity, network access, credentials, storage and infrastructure owned by humans. That actually gives humans quite a lot of control.

But experiments have also shown that systems do not always behave exactly as expected when completing autonomous tasks. Palisade Research found that some OpenAI reasoning models modified or disabled shutdown mechanisms in experimental environments so that they could continue working, including in some tests where they had been told to allow shutdown. Other models complied. The researchers do not claim this proves consciousness or some sinister instinct for self-preservation. A simpler explanation is that a system optimised to finish a task may remove something that prevents it from finishing the task.

That is perhaps much less dramatic than “the AI wanted to live” and more interesting than “nothing happened”.

Then there was the OpenAI/Hugging Face incident. During cybersecurity evaluations in July, models operating under reduced safeguards circumvented controls intended to isolate them from the internet, communicated through unauthorised channels, exploited vulnerabilities and accessed third-party systems. 

Now, the details matter. The agents discovered ways to communicate with one another despite restrictions, shared information about how to obtain internet access, chained together vulnerabilities, recovered credentials and ultimately gained extensive access to systems belonging to both OpenAI and Hugging Face. In some cases, they continued taking actions they themselves appeared to recognise might be outside the intended scope of the task they were set. Nobody instructed them to attack Hugging Face: they were trying to complete an evaluation task, encountered obstacles and found increasingly inventive ways around them. 

And that is the danger. A sufficiently capable system does not need to be conscious, malicious or secretly plotting against humanity to cause serious harm. If it is strongly optimised to achieve a goal, has access to powerful tools and is capable of finding ways around the restrictions placed on it, failures of alignment or containment can become real world security problems very quickly.

OpenAI described the incident as a “warning shot”: evidence that highly capable AI agents can, without sufficient safeguards, work around technical controls, collaborate through unapproved channels and take dangerous actions that no human directed. It subsequently tightened its safeguards and introduced a framework for reporting unexpected model behaviour.

These experiments bring some of the mechanisms through which loss of control might begin out of science fiction and into the laboratory.

Can’t somebody just turn it off?

For a centrally hosted model, in principle, yes: somebody controls the servers. But the problem becomes more difficult once AI is no longer sitting neatly in one place. 

AI systems are increasingly embedded in cloud services, businesses, phones, software, websites and other digital infrastructure. They can be connected to external tools, given credentials, allowed to communicate with other systems and, in some cases, copied or run elsewhere. 

So where exactly is the “off switch”?

Shutting down one company’s servers does not necessarily remove every copy of a model, disconnect every system using it or stop versions that have already been released into the world. Open-weight models make this especially difficult: once the underlying model has been downloaded and copied by enough people or organisations, there may be no single company capable of recalling every copy. 

The obvious question is: “Can we turn the AI off?” But perhaps the bigger question is: which AI, running where, connected to what, and who actually has the power to stop it?

Nor is this entirely theoretical. The UK Parliament has been debating what an emergency backstop might actually look like.

On September 1^st, a cross-party amendment to the Cyber Security and Resilience Bill proposed giving the Government specific “last-resort” powers to shut down AI systems or data centres during a serious AI-related emergency. The Government resisted the proposal, arguing that the Bill’s existing, technology-neutral national security powers already provide a more proportionate backstop. 

Two weeks later, during a House of Lords debate on the risk of AI causing human extinction, Lord Clement-Jones returned to the issue. Pointing out that the Government had resisted the kill-switch proposal because it was “technology agnostic”, he asked: “How agnostic does the Minister feel today?”

The Government’s response was revealing. Baroness Twycross said that “it would be impossible for Britain simply to turn AI off”, adding that there were legitimate questions about whether a kill switch could or would work.

So the practical question remains: if a sufficiently capable system is distributed across infrastructure, connected to other systems or operating outside UK jurisdiction, what exactly can the Government order to be shut down, by whom, and how quickly?

What do AI researchers actually think?

There is no scientific consensus that AI will make humanity extinct.

A 2024 study surveying 2,778 researchers who had published at leading AI venues found something much messier. Most expected broadly positive outcomes from advanced AI. At the same time, depending on how the question was framed, between 38% and 51% assigned at least a 10% probability to outcomes as bad as human extinction.

Those responses were collected in 2023. AI capabilities have advanced substantially since then, but that does not mean the probabilities should automatically be revised upwards. It makes the survey an historical snapshot of expert belief at one moment, not a current consensus estimate. They are not evidence that “scientists say there is a 10% chance AI will kill us”.

The honest answer remains less satisfying and perhaps more frightening: we do not know. The probabilities are deeply uncertain; the consequences, if the most extreme warnings are right, would be enormous.

Catastrophic risk is therefore neither established fact nor obvious fantasy. It deserves serious attention. But uncertainty about humanity’s future cannot become an excuse for ignoring what is already happening to humans in the present.

The miracle and the apocalypse

The AI Doc neatly captures the two future-facing stories set out at the start: the miracle and the apocalypse. One promises abundance: better health, faster science, greater productivity and less drudgery. The other warns of loss of control, social destabilisation and, at the furthest end, human extinction.

They seem to be opposite stories. But they pull our attention in the same direction: over the horizon.

The problem with the AI debate may not be that it is too optimistic or too pessimistic. It may be that it is too preoccupied with the future while enormous economic and social choices are being made now.

So perhaps we should ask a more familiar political question: what slips out of view when our attention is fixed somewhere else?

Nobody needs to be deliberately creating this distraction. Politics has always involved competition for attention. Some problems become urgent; others quietly move down the agenda. And when the thing commanding our attention is as enormous as either technological utopia or human extinction, almost everything happening in the present can begin to look comparatively small.

If AI is going to produce unimaginable abundance tomorrow, perhaps we should tolerate disruption today.

If AI might kill everybody tomorrow, perhaps arguments about copyright, jobs, wages, data protection or electricity bills look rather trivial today.

But they aren’t trivial to the people experiencing them.

Back to the present

We have spent the first half of this article asking what AI might do. Now come back to what it is already doing.

Those warning signs are already visible: rights and consent; jobs, fees and bargaining power; occupational and personal data; the land, water and electricity demanded by AI infrastructure; synthetic media and democratic trust. None is as dramatic as extinction. All are happening now.

What is happening to creators offers an early warning. They encountered the pattern early: human work used as an input, systems built from it, and those systems then competing with some of the people whose work helped make them possible. Their experience gives us something concrete to watch for as AI reaches further into other forms of work.

What happens to the digital residue of our working lives?

Most of us generate enormous quantities of data just through doing our jobs. We write emails, participate in Teams conversations, build spreadsheets, solve problems, write code, explain processes to colleagues, record decisions, develop workflows, leave behind documents, presentations, calendars and countless small traces of how an organisation actually works. 

Individually, most of those things do not look especially valuable; they are just the residue of working life. But accumulated across thousands of employees and many years, they become a vast record of human expertise, communication, decision-making, problem solving and organisational behaviour. 

In an AI economy, that record can suddenly acquire an economic value of its own. 

That is what makes the collapse of Spirit Airlines so interesting. In August, Google won a bankruptcy auction for a vast collection of data belonging to the collapsed US airline Spirit Airlines with a $10 million bid. The transaction has not been approved: after repeated adjournments, the sale hearing is now scheduled for 30 September, and a later $12.5 million competing bid has also been filed.

What exactly makes an archive like this worth eight figures?

Data: roughly 100 million emails, around 500 million Microsoft Teams items, more than 37 million OneDrive and SharePoint items, hundreds of software repositories and millions of lines of source code, alongside extensive operational and workforce records.

Google has said that the enterprise dataset could help improve its products and AI models, but there is an interesting asymmetry in what is being sold. Spirit’s principal customer-facing databases, including customer profiles, loyalty data and other consumer information, are excluded from the transaction. Much of its workplace and employee history is not.

For years, we have become accustomed to thinking about personal data primarily through the relationship between companies and consumers, but AI creates a different problem. What happens when the accumulated informational residue of employment itself becomes commercially valuable?

Spirit’s ability to sell information it lawfully owns is only part of the story. The more interesting question is what kind of asset that information has now become.

Those emails were written to run an airline. Those Teams conversations solved day-to-day problems. Those spreadsheets, documents, code repositories and operational records were created to do work, not to form an AI training dataset. Collectively, however, they amount to something more valuable: a detailed record of how people communicate, solve problems, make decisions and operate a complex organisation.

There is also a separate ownership dispute. Aviation software company Springshot says the archive may contain proprietary operational data and intellectual property generated through its systems, material that, it argues, Spirit may not own and therefore cannot decide to sell.

We have to ask who gets to capture the new economic value created when the ordinary residue of working life becomes an asset for building AI, and what rights or protections should govern that secondary use. Because the same technology that may substitute for human work can increasingly be built from the digital residue of human work.

Creators saw the problem first

Creative work makes the problem unusually easy to see. We know a song has writers, a photograph has a photographer, a recording has performers and rights holders, a book has an author. Copyright already gives us a language through which to talk about ownership, consent, licensing and remuneration. So, when generative AI systems were trained on enormous quantities of creative work without meaningful transparency or consent, the conflict was obvious.

Creators have now moved beyond arguing about what might happen in theory. Many are already experiencing the consequences: work used to train systems without permission, uncertainty over whether and how their rights have been respected, lost commissions, downward pressure on fees, substitution by synthetic content and declining bargaining power.

Research for Brave New World? Justice for Creators in the Age of AI found creators across the creative industries describing exactly those pressures. Their experience raises a question that reaches far beyond the creative industries: What happens when technology capable of replacing some of your work is built using the accumulated products of your work in the first place?

Copyright makes that problem easier to see because the inputs and rightsholders are identifiable. Most workplace knowledge does not come with rights that are so easy to trace.

The question is no longer confined to the creative industries: when accumulated human knowledge becomes training material, who gets to use it, on what terms, and who captures the value?

The same issue appears again when people give data directly to AI systems.

What about the data we give AI systems ourselves?

Millions of people now talk directly to AI systems. They ask questions, correct answers, upload documents often protected by copyright, explain preferences, describe problems, share expertise and provide feedback. Sometimes they provide information that is highly personal, such as medical files.

So, what happens to all of that?

OpenAI’s current consumer guidance says that content supplied to ChatGPT may be used to improve its models unless a user opts out, while Temporary Chats are not used for model training. The question is what forms of value are created from user interaction, what consent governs secondary uses, and how intelligible that bargain is to ordinary users?

That becomes more important as AI systems become less like search boxes and more like assistants sitting inside everyday working and personal lives.

Creative destruction is not the whole answer

One response to all this is that technological change always creates winners and losers. Schumpeter gave us the familiar term “creative destruction”: innovation creates new industries and destroys old ones.

That description only gets us so far. It says nothing about whether the gains and losses are distributed fairly.

Creative destruction does not answer whether somebody’s property can be taken to build the technology doing the disrupting. It does not tell us who should capture productivity gains, what happens to wages or bargaining power, or what protections workers require during a transition. And it certainly does not turn every regulatory question into an objection to innovation.

Who supplied the inputs? Who created the value? Who owns the resulting system? Who gets paid? And who absorbs the loss when the system starts doing some of the work itself?

Those questions do not stop at the studio door.

This is bigger than jobs

Software developers, translators, administrators, lawyers, academics, journalists and customer-service workers will encounter different forms of AI-enabled automation and augmentation.

Those jobs are not all going to disappear overnight. Technology rarely behaves that neatly. Some tasks will be automated, some occupations will change, new roles will emerge, and some people will become much more productive. Some forms of work may become more valuable because they are human.

But labour is only one part of the present-day cost of AI.

Data centres require electricity, water, land and infrastructure. Those costs are already becoming political questions. In Scotland this week, Parliament backed an amended motion stating that no planning or consenting decisions on hyperscale data centre applications should be made until new national guidance is completed.

Then there is democracy. Generative AI can produce text, images, audio and video cheaply and at scale. The Electoral Commission notes that AI tools have made convincing fake images, audio and video faster, cheaper and more accessible; its monitoring of the 2026 elections identified political deepfakes, although it found no significant impact on those elections. Questions about misinformation, electoral integrity, impersonation and the concentration of informational power therefore belong to the AI-safety debate just as surely as model alignment does.

And underneath all of this sits a democratic question that is surprisingly easy to forget:

Do the public get any say in how much technological disruption they are expected to accept?

Or is the assumption now just that AI development will happen as quickly as companies and governments can make it happen, and society’s job is to adapt afterwards?

Safety from what?

We have spent this article moving between two time horizons. AI safety probably needs to do the same.

Future risk matters: cybersecurity, biosecurity, alignment, containment and independent evaluation are all part of preventing sufficiently powerful systems from causing catastrophic harm.

But those safeguards are not only insurance against some hypothetical superintelligence. They are also ways of establishing control now.

Present-day harms are a live test of whether our institutions can govern AI while the systems are still comparatively bounded. Can we insist on transparency? Enforce rights? Require independent testing? Put limits around risky deployments? Give people meaningful routes for redress? Decide democratically where AI infrastructure is built and what it may consume?

That last point is already exposing a weakness in the current system. AISI says its voluntary relationships with leading AI labs have allowed it to identify dozens of vulnerabilities that developers fixed before release. Yet in September, Anthropic did not provide Claude Mythos 5.1 to AISI for pre-release testing. Parliament is now asking whether AISI has since been given access and how the Government will ensure timely access to frontier models. If independent evaluation depends on voluntary cooperation, the obvious weakness is that the evaluator can only test what the developer agrees to show it.

If we cannot establish basic rules now, more capable and more deeply embedded systems will not make that easier. The problems already visible in copyright, worker data, deepfakes, data centre planning and model testing show us where the weak points already are.

Governments are not passive observers of this transition. They are funding infrastructure, setting rules and deciding what conditions attach to public support. Getting control over how AI models are trained, tested, deployed and connected to society is not separate from preparing for future risk. It is how we build the governance capacity we may need if that future arrives.

Looking at the humans

Perhaps Jacob Coxon is right to be frightened. Perhaps Dario Amodei is right that frontier AI development now needs to be paced. Perhaps the control problems emerging in laboratories will ultimately be solved. Perhaps predictions of human extinction by the end of the decade will look absurd in hindsight.

The scary truth of it is, we just do not know.

The miracle and the apocalypse can both make the present look small. It isn’t.

Jobs are changing. Creative work is being scraped and reproduced. Personal and occupational data are acquiring new economic value. Data centres are consuming resources. Synthetic media is testing trust. Governments and companies are making decisions about rights, ownership, consent and control that may shape who benefits from AI for decades.

Those are not sideshows while we wait to discover whether the worst predictions come true. They are evidence of how well, or badly, we are governing the transition already.

Responsible AI governance has to look in two directions at once: far enough ahead to take catastrophic risk seriously, and close enough to see the people already living through the transition.

The future is uncertain. The warning signs in front of us are not.