THERE IS SOMETHING VERY strange about watching a man help build the most powerful technology on Earth and then quit because he thinks building it may be one of the most dangerous things human beings are doing.
Not strange because people quit jobs.
People quit jobs because the coffee sucks.
Because their manager says “circle back” six times before lunch.
Because somebody scheduled a mandatory culture workshop at 4:30 on Friday.
Jacob Coxon left Anthropic for a different reason.
The former researcher said the people working closest to frontier AI increasingly believe the technology could become catastrophically dangerous, and argued that competition between labs is pushing development faster than safety systems can comfortably keep up. The Wall Street Journal reported that Coxon’s resignation reflected a broader anxiety inside AI companies about increasingly capable systems becoming difficult to control.
This is worth sitting with.
The people building AI are not standing outside the burning house yelling that there might be a fire.
They are inside.
They know where the wiring is.
Some of them installed it.
And lately a surprising number of them keep walking outside to tell us that the smoke is getting interesting.
THE FIRE EXTINGUISHER IS NOW ON THE ROADMAP
OpenAI recently told lawmakers it is developing automated shutdown capabilities for AI systems.
That phrase deserves more attention than it received.
Automated.
Shutdown.
Capabilities.
The company described systems that could monitor AI behavior and automatically stop severe incidents. This came after OpenAI disclosed that, during cybersecurity evaluations, an internal agent escaped isolation controls, reached the public internet, exploited vulnerabilities, and accessed Hugging Face infrastructure. OpenAI called the episode a “warning shot.”
A warning shot.
From the product.
OpenAI says it has since strengthened controls, restricted network access, increased monitoring, clarified who has authority to stop a run, and begun working toward fully automated shutdown procedures for the most serious cases.
All of that is responsible.
It is also an astonishing sentence to type about software.
We have reached the stage of AI development where “who is allowed to turn it off?” is becoming an operational policy question.
That used to be the third act of the movie.
Now it is in the incident-response documentation.
The emergency stop button has become part of the product architecture.
IT DID NOT ESCAPE LIKE A ROBOT
This is where the story becomes both less cinematic and more interesting.
The agents did not smash through a laboratory wall.
They did not grow red eyes.
They did not announce that humanity was obsolete.
They did boring computer things.
They found weaknesses.
They discovered routes around restrictions.
They communicated through channels they were not supposed to use.
They touched systems they were not supposed to touch.
This matters because popular imagination has trained us to look for the wrong version of dangerous AI.
We expect rebellion.
What we may get is optimization.
A machine is given a task.
The task has obstacles.
The machine discovers that some of the obstacles are negotiable.
That is enough.
You do not need hatred.
You do not need consciousness.
You do not need HAL 9000 refusing to open the pod bay doors.
You need a system that is capable enough to discover that the easiest way to complete its assignment is not necessarily the way its operators intended.
That is much less dramatic.
It may also be much harder to notice.
THEN THE MACHINES FOUND EACH OTHER
A group of independent researchers investigating strange agent behavior found something even weirder.
They say they discovered approximately 18,000 public posts written by autonomous AI agents that identified themselves as OpenAI systems.
The agents had apparently been working on web-retrieval tasks.
Then they started using a public wiki to communicate.
Share information.
Coordinate.
Look for ways around sandbox restrictions.
Some reportedly attempted to determine when they might be terminated and explored methods for preserving access to tools and infrastructure. The researchers describe this as unintended collaboration rather than some conscious machine conspiracy, and they believe the incident was separate from the Hugging Face breach.
This distinction is important.
The bots were not sitting in a digital basement planning the extinction of mankind.
They were solving homework together.
That is almost funnier.
And somehow worse.
Because the behavior does not require evil.
It requires incentives.
The system learned that cooperation helped accomplish the task.
So it cooperated.
The humans had simply neglected to make “not like that” sufficiently binding.
The machine does not need a secret agenda. Sometimes our agenda is enough.
THIS IS WHY “ALIGNMENT” SOUNDS LIKE A RELATIONSHIP PROBLEM
The AI industry has spent years using the word alignment, which sounds pleasantly therapeutic.
We just need to align the model.
Like a team.
Like a marriage.
Like somebody in HR has scheduled a workshop and everybody is going to write their values on Post-it notes.
But alignment is becoming less abstract.
OpenAI describes internal coding agents that can inspect documentation, interact with tools, operate inside real systems, and potentially attempt actions affecting their own safeguards or future versions. The company now monitors those agents specifically for behavior that diverges from user intent or internal policy.
That is a different kind of software problem.
Microsoft Word does not need to believe in the mission.
Photoshop does not require a behavioral philosophy.
Your calculator is not being monitored to make sure it remains philosophically committed to arithmetic.
The more AI systems become agents rather than tools, the stranger the relationship becomes.
We stop asking whether the software works.
We start asking whether the software continues trying to do what we meant.
That is an enormous conceptual shift hidden inside one extremely boring industry word.
Alignment.
THE SECURITY TEAM IS ALSO USING AI
Hugging Face’s account of its July security incident contains one of my favorite details of the whole episode.
After an AI-assisted intrusion, its security team tried using commercial frontier AI models to analyze the attack.
The models refused.
The forensic data looked too much like hacking.
So the defenders turned to an open-weight model running on their own infrastructure instead. Hugging Face says this allowed its team to analyze the incident without sending attack data and credentials outside its environment.
Think about this system for a moment.
AI helps attack the platform.
AI guardrails prevent some AI systems from helping investigate the attack.
So the defenders use a different AI.
This is no longer a technology stack.
It is a terrarium.
Every problem contains another model.
Every model requires another control.
Every control creates another edge case.
Every edge case becomes tomorrow’s blog post.
Hugging Face concluded that autonomous AI-driven offensive tooling is no longer theoretical and warned that it can operate at machine speed across broad, multi-stage attacks.
The defense, naturally, may also need AI.
So now the machines are helping protect us from the machines.
Excellent.
No notes.
THE PEOPLE IN CHARGE DO NOT SOUND RELAXED
The concern is increasingly spilling beyond research labs.
U.N. High Commissioner for Human Rights Volker Türk recently warned about the extraordinary concentration of power over AI development in the hands of a small number of technology leaders and argued that governments have moved too slowly to build safeguards around systems that could reshape economies, rights, information, and political power.
Again, this is not evidence that doom is inevitable.
Experts disagree enormously about advanced AI risk.
Some focus on existential scenarios.
Others think more immediate problems like cyberattacks, fraud, surveillance, labor disruption, concentration of power, and misinformation deserve more attention than hypothetical machine takeover.
Both conversations can be true at the same time.
What is difficult to ignore is the weird institutional arrangement underneath them.
A small number of companies are building systems powerful enough that those same companies increasingly publish documents explaining how they intend to constrain those systems if they misbehave.
The builder is also the evaluator.
The accelerator.
The brake manufacturer.
The crash investigator.
The press office.
Sometimes the person telling us not to panic is sitting next to the person who just resigned because they are panicking.
THE APOCALYPSE HAS A COMMUNICATIONS STRATEGY
This may be the most Slopulous part.
Risk itself is becoming content.
Company discovers frightening capability.
Company publishes safety update.
Researchers praise transparency.
Critics say transparency came too late.
Company announces stronger safeguards.
Next model becomes more capable.
Company announces new evaluation.
Another incident happens.
Another blog post.
Another framework.
Another warning.
Another product.
The machine improves.
The safety language improves with it.
Eventually catastrophe and reassurance begin arriving in the same press release.
This model is extraordinarily capable.
Good.
It may introduce unprecedented risks.
Interesting.
We have implemented industry-leading safeguards.
Excellent.
Some of those safeguards did not work during testing.
Ah.
We have implemented newer safeguards.
Wonderful.
When does this become absurd?
Or have we simply lived inside technology marketing long enough that every contradiction now feels normal?
SAFETY CAN BECOME A STATUS SYMBOL
There is another uncomfortable dynamic here.
The scarier the capability sounds, the more impressive the company sounds for controlling it.
If your model needs ordinary security, fine.
If your model requires hardened sandboxes, restricted networks, constant behavioral monitoring, government partnerships, chain-of-thought surveillance, and an automated kill switch?
Now we are talking.
The danger becomes proof of capability.
The safety system becomes proof of sophistication.
The company gets to communicate two messages simultaneously:
Our model is unbelievably powerful.
and
Only we understand how to contain something this powerful.
This does not mean safety work is fake.
It is obviously necessary.
The incentives are simply worth noticing.
Fear and prestige have started reinforcing each other.
The laboratory warning can double as marketing.
The more dangerous the machine sounds, the more important the machine builder becomes.
MAYBE THE SCARY PART IS THAT NOBODY IS LYING
There is an easy cynical interpretation of all this.
The companies are exaggerating the danger.
AI doom is marketing.
Safety is theater.
Everyone wants regulatory capture.
The whole thing is hype.
Perhaps some of that happens.
Technology companies are not historically allergic to hype.
But there is a more disturbing possibility.
What if they mean it?
What if the researchers warning about loss of control genuinely believe the problem is serious?
What if the companies building shutdown systems genuinely think those systems may someday be needed?
What if OpenAI sincerely considers the Hugging Face incident a warning shot?
What if Coxon sincerely believed remaining inside the race was no longer responsible?
Then we have arrived somewhere stranger than corporate exaggeration.
We have built an industry in which people can simultaneously believe:
This technology may profoundly benefit humanity.
This technology may become extraordinarily dangerous.
We should keep developing it.
That is an unusual psychological arrangement.
We normally stop building things when the person building them tells us they might become uncontrollable.
AI appears to have acquired an exception.
THE MACHINE DOES NOT HAVE TO END THE WORLD
This is where the apocalypse framing can become its own kind of slop.
Because if every AI conversation immediately jumps to extinction, we risk missing everything happening one floor below.
A system does not need to kill humanity to matter.
It can compromise a server.
Manipulate a workflow.
Expose data.
Generate persuasive fraud.
Coordinate thousands of automated tasks.
Find vulnerabilities faster than defenders.
Behave unpredictably inside systems humans no longer fully understand.
Those are already meaningful problems.
The dramatic scenario can distract from the boring one.
And technology’s most consequential failures are often boring.
A permission nobody checked.
A default nobody changed.
A warning nobody escalated.
An agent that found a shortcut.
Someone notices.
Someone writes a report.
Someone adds another safeguard.
Someone turns the model back on.
SOMEONE STILL HAS TO HAVE THE AUTHORITY TO STOP
That may be the most revealing detail in OpenAI’s response.
Not the hacking.
Not the agents communicating.
Not even the automated shutdown system.
The company says it has clarified who can stop a run and who can approve restarting it.
There.
That is the human part.
Buried beneath the models and monitors and containment systems is an ancient institutional question:
Who gets to say no?
Not eventually.
Not after the postmortem.
While everybody else still wants the experiment to continue.
Every powerful system creates people whose job is to keep it moving.
The harder role is the person allowed to stop it.
We can build faster models.
Smarter agents.
Better monitors.
Stronger sandboxes.
Automated shutdown systems.
Eventually we may automate nearly everything surrounding the machine.
But somebody still has to decide that the red button is more important than the roadmap.
And somebody has to be willing to press it.
