Safety Is the Name of the Centralization

The strongest objection to this entire series is not that self-hosting is expensive, or that the frontier stays ahead, or that I moved my dependencies without removing them. I have spent nine essays conceding those. The strongest objection is the one that says the thing I am defending should not exist. It goes like this: sufficiently capable models are dangerous in a way that cannot be taken back, open weights spread that danger to anyone who downloads them, and the only place you can actually hold the line is a small number of closed systems watched closely by people who can pull the plug. On that account, my desk machine running open weights is not sovereignty. It is a hole in the fence.

I want to give that argument its full weight before I answer it, because it is the best argument against me. And then I want to show what it is actually asking for, which is something the people making it rarely say out loud.

The strongest safety case, steelmanned

Start with the people who signed their names to it. In 2023 the Center for AI Safety published a one-sentence statement that read, in full: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” The signatories were not cranks. They included Geoffrey Hinton and Yoshua Bengio, two of the three men who won the Turing Award for the deep learning that made all of this possible, alongside the chief executives of the leading frontier labs themselves. When the people building the most capable systems on earth sign a statement comparing their own work to nuclear war, the honest move is not to laugh. It is to ask what they think they see.

What they think they see is irreversibility. This is the part the open-weights camp, my camp, has the hardest time answering, so let me state it at full strength. The UK government’s AI Safety Institute put it plainly in its analysis of open-weight risk: once weights are released, harmful capability can “proliferate rapidly and irreversibly,” and released models are “extremely vulnerable to adversarial detuning.” Read those two clauses carefully, because they are not rhetoric. They are mechanically true.

You cannot un-release a weight file. There is no recall, no patch you can force onto a million copies already on a million disks. A closed model behind an API can be corrected overnight: the provider changes the system prompt, retrains the refusal behavior, revokes a key, and every user is moved at once. An open weight has no such lever, because the whole point of an open weight is that no one holds the lever. And the safety training that makes a model decline the dangerous request is, on current evidence, cheap to strip. Fine-tuning the refusals back out costs a tiny fraction of what training the model cost in the first place. So the worry is not that someone downloads a dangerous model. It is that someone downloads a safe one and spends an afternoon making it dangerous, and there is no version of “we fixed it” that reaches them.

I do not think that argument is wrong. I think it is the load-bearing reason a serious person can favor closed systems, and any sovereignty writer who waves it away is selling you something. Hold it in your head at full strength. Now I am going to show you what it costs.

The counter the incumbents do not want named

There is a second tradition, equally serious, that looks at the same statement and the same institutes and sees something other than disinterested caution. Yann LeCun, the third Turing Award winner, the one who did not sign, has called the existential framing a “Doomer’s Delusion” and read the safety-driven push for regulation as, in effect, a bid for power: a way to frighten governments into rules that only the largest incumbents can afford to satisfy. You do not have to take his civilizational optimism on faith to notice that he is pointing at a real structure.

Andrew Ng, who built much of the practical machine learning the field runs on, has been sharper about the mechanism. In his written statement to the U.S. Senate AI Insight Forum, he warned that overblown fears of catastrophe are being used to justify regulation that would entrench dominant firms and crush open-source and smaller players. This is the classic shape of regulatory capture: the cost of compliance is fixed, so it falls hardest on the small. A licensing regime, a registration threshold, a mandate that every capable model run inside an approved monitoring layer, each of these is trivial for a company with a legal department and a billion-dollar compute budget, and fatal for a person with a machine on a desk. The rule does not have to name the open-source operator as the enemy. It only has to price him out.

And the centralizing logic is not hiding. The Council on Foreign Relations, no fringe outlet, frames the coming year as a contest decided by aggregate compute, with export controls described as the only tool capable of slowing a rival’s progress, and observes that a “state-centric model could prove better suited to deploying autonomous systems at scale.” That is the establishment saying the quiet part in policy language: the frontier is won by whoever concentrates the most compute, and concentration is the natural order of things. The CFR is not making a safety argument. It is making a power argument. But the two arguments ask for exactly the same thing.

The press was dangerous too, and they licensed it

None of this is new. We have run the experiment before, on the last information technology that frightened the people in charge.

When the printing press spread through Europe, authorities did not see a tool. They saw a hazard, and they moved to control it in the name of order and orthodoxy. In England the answer took a familiar shape. The Stationers’ Company received a royal charter in 1557 granting it a monopoly over printing, and through the Star Chamber and later the Licensing Order of 1643 the state required pre-publication licensing: official approval before anything could lawfully be printed. The justification was safety in the only vocabulary that era had for it. Unlicensed printing was said to spread sedition and heresy, threats to the realm and to the church, and so the remedy was to permit only approved hands to operate the press. John Milton answered that regime directly in Areopagitica in 1644, arguing against licensing before publication rather than against printing itself.

Hold the structure up against the present one. A powerful new way to produce and spread information appears. It is treated as dangerous in a way that demands a response. The response is not to ban the technology, which no one could manage anyway, but to control who is permitted to operate it, justified as safety, which quietly concentrates the power to print in licensed hands. The licenser was never against the printed word. He was against the unlicensed press, which is a different thing, and the difference was the whole point.

“Monitor and gate at the inference layer for safety” is that licensing order rewritten for models. The same structural move, the same justification, the same concentration. The technology changed. The argument did not, and neither did what it asks of the person who wanted to operate the machine without first asking.

Monitor at inference, control who computes

Here is the structural claim, and it is the spine of this essay.

The safety proposal, in its most reasonable form, is not “ban capable models.” Almost no serious person says that. The reasonable form is: keep the most capable models behind an inference layer that can watch what is being asked of them, refuse the dangerous request, log the pattern, and revoke access from the actor who keeps trying. Monitor at the point of inference. That is the proposal. It sounds like a smoke detector. It is not a smoke detector.

To monitor every inference, you need every inference to pass through a place you control. And the only way to guarantee that every inference passes through a place you control is to make running the model anywhere else either impossible or illegal. There is no version of “we watch all the inference” that does not also mean “no one computes outside the watched channel.” The monitoring layer and the permission layer are the same layer. You cannot have the first without building the second, because an unmonitored machine is, by definition, the exact thing the proposal exists to prevent. So “monitor capability at the inference layer” resolves, structurally and without anyone having to intend it, into “control who is allowed to compute.” Strip the gentler phrasing and what remains is a permit to think with a machine, granted by whoever holds the lever and revocable at their discretion. Those are not two policies. They are one policy described at two levels of honesty.

And once you see that, the question stops being “is monitoring good?” and becomes “who holds the monitoring lever, and what happens when they are wrong?” A lever that can switch off the bad actor is the same physical lever that can switch off the dissident, the competitor, the journalist, the person in the wrong country, the person whose use the controller simply does not like. The lever does not know the difference. It only knows on and off. We have a long record of what happens to infrastructure built to stop the worst people once it exists: it gets pointed at ordinary people, because the people holding it always discover new categories of threat that happen to be convenient. The proposal asks me to trust that the one institution with a kill switch over all computation will only ever use it on the genuinely dangerous. I do not have to believe that institution is evil to decline that bet. I only have to believe it is an institution.

This is the move I have made in an earlier essay: you cannot remove a dependency, you can only choose whether it is one you can see and stand on. The centralizing safety proposal removes that choice for everyone at once. It does not relocate the dependency. It abolishes the alternative.

What “control who computes” forecloses for one operator

Let me make this concrete instead of abstract, because the abstract version lets everyone imagine it lands on someone else.

I run open weights on hardware I own, on a desk: a model I can read, a sampler I can change, logs that stay on my disk, an inference path I do not have to ask anyone’s permission to run. The supply-chain dimension of that setup is precisely what a “control who computes” regime forecloses. Today the weights arrive as a file I am allowed to download, hold, and serve. A monitoring mandate does not have to confiscate my machine to end that. It only has to make the capable open weight unavailable, or make serving it without an approved monitoring layer a violation, and the file simply stops arriving. The hardware on my desk becomes a box that can only legally run models that phone home. No one needs to confiscate a press they have made it illegal to ink. I would not be raided. I would be deprecated. The capability would still exist, in full, inside the watched channel, available to anyone who agrees to be watched, and nowhere else.

That is the actual stake. Not my comfort, not my hobby, but whether a single person can hold and run a general capability without a larger party’s standing permission. The closed-frontier-plus-monitoring world is one where the answer is no, structurally, for everyone not large enough to be the monitor. And I have a stake in that answer, so let me say it instead of pretending neutrality: I run the thing the centralizing move would foreclose. My defense of it is not disinterested. The honesty stance this site is built on requires me to put that on the table rather than launder my position as pure principle. I am arguing for a world in which my own machine stays legal. That does not make the argument wrong. It makes me obligated to tell you I am in it.

What I am actually claiming

I am not claiming the proliferation worry is fake. I have steelmanned it because I believe it. You cannot un-release a weight, safety can be detuned for the price of an afternoon, and somewhere in the space of possible capabilities there is almost certainly something that should not be a free download. If you came here for a writer who tells you the danger is imaginary, I am not him. The residual risk is real and it does not go away because the centralizing cure is worse. Both things are true at once, and the honest position is the uncomfortable one that holds both.

What I am claiming is narrower and harder to dismiss. The proposal to manage that real danger by monitoring capability at the inference layer is not a smaller move than controlling who is allowed to compute. It is the same move, described more gently. And the concentration of that control, a single lever over all general computation held by whichever party is large enough to be trusted with it, is a more durable and less correctable danger than the proliferation it is meant to prevent. A misused open weight is a bounded harm by a bounded actor. A misused kill switch over all computation is unbounded, and it has no off switch of its own, because the only party who could pull it is the party holding it. The comparison table at the top of this essay is that distinction unrolled: the same word, safety, naming two opposite architectures, and the entire fight is over which one we let it mean.

Distributed resilience is not the safer-sounding option. It is the genuinely riskier one in the short run, and I am conceding that on the record. It means more hands on more capability and no central authority who can fix a mistake for everyone overnight. What it buys, in exchange for that risk, is that no single party can decide who counts as dangerous, and that when the controller is wrong about you, and controllers are always eventually wrong about someone, you keep operating. I will take a world full of fallible operators over a world with one infallible warden, because I do not believe the warden is infallible, and neither, when they signed that statement, did the people who built him.

This closes the first arc of the series. Ten essays, each conceding its strongest objection before answering it, from the radical monopoly of the rented model through to this one, where the objection was not about cost or capability but about whether the thing should be permitted at all. The spine of the whole argument, and the order in which the pieces sit, lives on the philosophy page. The structured, complete version is the forthcoming book, for which these essays are the public workshop. The first arc asked what sovereignty is and what it costs. The next one will ask what it is for.