Too Dangerous to Share

Too Dangerous to Share

Written on 13 September 2026.

AI may become too dangerous for governments to want everyone to possess it. But wanting to control it and being able to control it are different things. The same technology that requires enormous organizations to create it may produce something that can be copied and used outside those organizations.

That changes the argument about centralized AI. It is possible that increasingly powerful systems will be kept inside a small number of approved institutions. But there is also a technical reason why that arrangement might fail: a large part of a trained model exists as digital information.

I initially thought the comparison with nuclear technology and high-containment biological laboratories made the direction fairly clear. More power would require more security, more infrastructure and more government involvement. Ordinary people would receive controlled access to services while the strongest capabilities remained elsewhere.

I still think that is a serious possibility. But model weights, open releases and specialized inference chips introduce an opposing pressure. It would be a mistake to explain that pressure away just to preserve a darker conclusion.

A slowdown does not necessarily stop the race

In his September 2026 essay, We Must Pace the Frontier, Dario Amodei calls for slowing capability development enough for safety work to keep up. He points to AI increasingly helping build subsequent AI systems. His proposals include outside evaluators, coordination between companies and governments, protection of model weights, and restrictions on advanced chips and unauthorized distillation by companies in authoritarian countries. He also says pacing does not mean stopping technical progress and must take account of competition with China.

There is a tension here. A government can believe that advanced AI is dangerous while also believing that another country must not develop it first. The danger then becomes a reason to restrict distribution without becoming a sufficient reason to abandon development.

This is where the Manhattan Project comparison is useful. It describes an incentive to continue strategically important research inside protected institutions. It does not establish that any particular secret AI project exists.

The question becomes who is permitted to continue. A small developer might face requirements he cannot afford, while a large laboratory with government support continues working. A slowdown could change participation more than it changes the destination.

Nor does that require the safety concerns to be invented. A company could sincerely fear dangerous capabilities and also benefit from regulations that make competition harder. Good intentions do not prevent a rule from concentrating power.

Recursive self-improvement also needs some care as a term. A system helping researchers write code or run experiments is not the same as a system independently creating a superior successor, repeatedly, with accelerating results. Amodei's warning is his assessment of an emerging process, not proof that unlimited improvement is inevitable.

Why the containment comparison matters

Biosafety Level 4 is the highest of the four conventional biosafety containment levels. The comparison with AI concerns the institutional requirements surrounding dangerous capabilities: specialized facilities, restricted access, trained personnel and oversight.

Anthropic itself made this connection. Its original Responsible Scaling Policy, introduced in September 2023, explicitly described its AI Safety Levels as loosely modeled on US biosafety standards. Greater potential danger was supposed to require stronger safeguards.

These company standards are not literal biological classifications or proof that all AI will become licensed technology. Anthropic has revised its policy since then, including its approach to coordinating safeguards beyond one company.

But the possible direction is clear. As the requirements become more demanding, fewer organizations may be able to satisfy them. Nobody has to announce that independent development is forbidden. It can become impractical through the conditions attached to it.

That remains an argument for centralization. The question is whether it also establishes lasting control over the product once the product exists.

What the weights contain

The weights of a neural network are the numerical parameters adjusted during training. Roughly speaking, they determine how strongly signals influence later computations. They are part of what makes a trained model behave differently from the same architecture before training.

They are not simply a collection of instructions describing how somebody else could train the model. They are a major part of the result of that training.

A useful model package also needs the right architecture, configuration, software and, for text models, tokenizer. The weights alone may not reproduce an entire commercial service, which can include search, external tools and other systems. But with the necessary supporting components, they can allow the trained network to run without repeating its original training process.

The size can be substantial without being beyond ordinary digital storage. As a simple calculation, one trillion parameters stored at two bytes each would occupy about two trillion bytes, or 2 TB, before additional information and overhead. That is an illustration, not a claim about the size of any particular closed model. Storage size also does not tell us how much computing power is needed to run it.

But it shows the important distinction. A training operation involving enormous facilities can leave behind numerical data that fits on a storage device. The device does not contain the facilities. It contains a reusable result of their work.

Expensive to create, easier to copy

Imagine that an institution spends billions developing a model. Someone else later obtains a usable copy of its trained parameters and the supporting information needed to run it.

The second party still needs hardware. It may need considerable engineering work. But it does not necessarily need to repeat the research and computation that originally produced the model.

This is where the nuclear comparison becomes incomplete. Obtaining the design of a nuclear device does not give someone the required nuclear material or the facilities to produce it. Obtaining a model's weights can provide a much larger fraction of the finished usable artifact. The remaining hardware requirement matters, but it is a different requirement from recreating the entire development process.

Digital information is not destined to leak. Strong security can prevent particular disclosures. But copying has a peculiar consequence: the original owner can retain everything while somebody else gains a copy. Nothing needs to disappear from the original installation.

Once usable copies spread, the original institution may find it much harder to restore exclusive possession. This does not make every leak public or every public copy easy to use. It means that control over creation and control over subsequent use can come apart.

There are also routes to wider access that do not involve a leak. A laboratory can deliberately release a model. Another laboratory can independently develop a competitive one. Distillation can train a different model using outputs from a stronger system, although it does not automatically reproduce every capability. These are different processes, with different limitations and permissions.

So the case for decentralization should not rest entirely on the assumption that somebody will steal something. The broader point is that useful capabilities can travel beyond the institution that first established them.

What specialized silicon changes

Taalas makes the issue more interesting because it targets the cost of running trained models. Its approach builds hardware around a particular model rather than relying entirely on general-purpose computation.

In its published HC1 demonstration, Taalas reported about 17,000 tokens per second per user for a hard-wired Llama 3.1 8B model. It also acknowledged quality loss from aggressive quantization. Those are company-reported results for that model, not evidence that any frontier model can already run at the same speed on an affordable desktop card.

AMD announced an agreement to acquire Taalas on 6 August 2026. Its announcement describes plans to integrate the technology into its accelerator roadmap and systems using Instinct GPUs. It does not announce the promised consumer card containing frontier intelligence.

Still, the possible consequence deserves attention. If specialized hardware makes a capable model much cheaper to run, possession of the model becomes more valuable. A copy that initially requires expensive equipment might later become practical for many more users.

That is a conditional argument about where the technology could go. Manufacturing a specialized chip still requires design work, fabrication and investment. It is not a matter of placing an arbitrary weight file onto a blank card. And a very fast small model does not automatically become a much smarter model merely by answering repeatedly.

More attempts, tools and checking can help when the system can recognize useful progress. They can also repeat mistakes. Speed expands what is practical; it does not guarantee frontier judgment.

But if the underlying model is already good enough, cheaper inference could be decisive. An individual does not have to own the facilities that trained it to benefit from running it privately.

The model need not come from the leading American laboratory

Suppose a Chinese laboratory, or a laboratory elsewhere, releases a sufficiently capable model. A hardware company could explore building an efficient implementation around it, subject to technical compatibility and the rights and permissions involved.

The model would not have to be the strongest in every benchmark. It would have to be good enough for the work people actually want to do.

That is a stronger argument than declaring that Chinese AI is simply better at everything or that one chip announcement makes the closed laboratories obsolete. Those sweeping claims are unnecessary. A competitive alternative can undermine dependence without winning every comparison.

Nor should Chinese development be reduced to copying American work. Independent research, deliberate open releases, distillation and unauthorized copying are separate possibilities. Which one occurred requires evidence in each case.

The practical question is whether capable models remain available from enough independent sources that no single provider can determine everyone's access. If they do, specialized hardware could increase that independence. If they do not, cheaper hardware alone cannot supply a model that nobody is willing or able to provide.

Can regulation prevent it?

A government could seek restrictions on particular models or hardware if it considered them dangerous. But a specific prohibition would require an applicable legal basis; saying national security is involved does not, by itself, establish that any proposed ban already exists or is enforceable everywhere.

AMD's own announcement identifies government measures, licensing requirements and reliance on outside manufacturers among the risks affecting its business. Physical manufacturing therefore remains a place where political restrictions could matter, even when the model files themselves are widely available.

But restricting one manufacturer is not the same as eliminating a capability internationally. Another jurisdiction might make a different decision. Another manufacturer might pursue a different implementation. Conversely, moving abroad would not automatically remove every legal or supply-chain obstacle.

This is a struggle over how much control can actually be exercised. We should not assume either perfect enforcement or complete helplessness.

There is also a difference between a leak benefiting another government and a leak benefiting ordinary people. If copied weights move from one protected laboratory into another, power may simply be redistributed between institutions. Wider individual freedom requires an additional step: practical access to a usable system.

That is why the inference question matters so much. The files and the ability to use them have to come together.

What this does to Kaczynski's argument

In paragraph 208 of Industrial Society and Its Future, Ted Kaczynski distinguishes technology usable by small communities without outside assistance from technology dependent on large-scale social organization.

That distinction still applies to semiconductor manufacturing and frontier model development. A person running AI at home has not independently reproduced the industrial system that made his equipment possible.

But it does not follow that this person must remain dependent on a central provider's permission for every use. Dependence on industrial production and dependence on an ongoing controlled service are different conditions.

I do not need to manufacture a computer myself to exercise meaningful control over software running on it. Similarly, I might not need to train a powerful model myself to gain meaningful independence by possessing and running it.

This is a real objection to a one-directional application of Kaczynski's argument. The same technological system can concentrate production and distribute useful capabilities. More advanced technology can introduce new dependencies while also removing particular forms of dependence.

So the GNU comparison cannot simply be dismissed. The free-software tradition recognizes something important about reproducible information: once people have the necessary files and the means to use them, the original producer's continuing control can be reduced. Open weights do not automatically provide every freedom associated with free software, but the copying principle remains relevant.

That does not refute everything Kaczynski argued about industrial society. It does weaken the claim that increasingly centralized development must lead to increasingly centralized possession of intelligence.

Good enough can be enough

There is another weakness in making the absolute frontier the only measure of freedom. A government might possess a much stronger system than an individual, while the individual's local model still gives him substantial independence in writing, programming, research and ordinary problem-solving.

Being behind does not make the capability worthless. A model that performs a person's work reliably does not necessarily stop being useful because a laboratory announces a better one.

This matters for the dream-carrot argument. The promise of everyone possessing the very latest intelligence may be exaggerated. But practical local independence does not require equality with every government laboratory. It requires enough useful capability under the person's own control.

The darker possibility remains: powerful institutions improve faster, contain their models successfully and restrict the hardware needed to run alternatives. Another possibility is that open releases, copied weights and falling inference costs repeatedly undermine that control. Neither follows automatically from the fact that frontier development is expensive.

I think the revised question is whether institutions can keep useful machine intelligence scarce after they have created it. They may control the largest training operations and still fail to preserve exclusive access to the results.

AI can be born inside an enormous centralized system without every useful copy remaining inside that system. That is a technological reason to question permanent centralized control, not merely a hopeful exception added to soften the conclusion.

References

AI Disclosure: Parts of this page may have been created, edited, or assisted by artificial intelligence tools (such as ChatGPT or other language models). All AI-assisted content is reviewed by a human before publication. For questions, contact the site administrator.