Who Should Govern AI?

"Who should govern AI?" sounds like a seminar question, the kind panels chew on while the real decisions get made somewhere else. But it is being answered right now, mostly by default. Every frontier model that ships, every capability that gets released or held back, every default setting on a product used by hundreds of millions of people — each is a governance decision, and today most of them are made by whoever got there first.

Maybe that default is even defensible. But it should be a choice, not an accident. So instead of picking a team, it's worth mapping the actual candidates for the job — what each one can genuinely do, and where each one runs out of reach.

The labs, governing themselves

The first candidate is already at work. The major AI labs have published safety frameworks — Anthropic calls its version a responsible scaling policy; its competitors have close cousins under other names. The shared idea is straightforward: define capability levels that would be dangerous, test each new model against them before release, and pre-commit to stronger safeguards, or to pausing, when a model crosses a line.

Give this its due. The labs employ most of the people on Earth who deeply understand these systems. They can act in weeks rather than years, and their frameworks have pushed real practices — red-teaming, staged releases, external testing before launch — into something like industry norms. No other candidate can match that expertise or speed.

The problems are just as plain. A safety framework is a promise a company made to itself, and a company can revise its own promises — especially when a competitor is about to ship. No outside body audits the evaluations. No penalty attaches to quietly softening a commitment. And the conflict of interest is structural, not a matter of character: the organization deciding whether a model is safe to release is the same one that spent enormous sums building it. Industries left to regulate themselves — finance, pharmaceuticals before modern drug agencies — tend to follow a familiar arc. Voluntary restraint holds until it becomes expensive.

Nations, writing law

The second candidate is national regulation, and three broad models are on the table.

The European Union's AI Act is the deepest attempt anywhere: the first comprehensive AI law from a major jurisdiction. Its core move is sorting by risk. A few uses are banned outright, such as social scoring by governments. High-risk uses — AI in hiring, credit decisions, medical devices, critical infrastructure — carry heavy obligations around testing, documentation, and human oversight. Lighter transparency duties cover the middle, and most everyday uses face nothing at all. A late addition, forced by the arrival of powerful general-purpose models mid-drafting, regulates the models themselves, with extra duties for the largest. The obligations phase in over several years rather than landing at once.

The criticisms are serious. Compliance costs fall hardest on smaller firms, which sits awkwardly with Europe's hopes of building its own AI industry. And because global companies tend to build one product for all markets, the Act functions as a regulatory export — rules the rest of the world never voted on but ends up living under. Admirers call this the Brussels effect and consider it a feature.

The United States has gone lighter and looser: no comprehensive federal statute, executive actions that shift with administrations, sector regulators stretching decades-old authority to cover new tools, and a growing patchwork of state laws. It's flexible, and it avoids freezing premature rules into place. It also means no one is clearly in charge.

China runs a third model: state-directed governance with registration requirements, licensing, and content controls built in from the start. It moves fast and reaches deep, but its aims — political stability prominent among them — differ from anything a democracy would choose.

Notice what all national rules share. They can reach deployment inside their borders — products, uses, harms to their own citizens. They cannot reach a training run on another continent, model weights crossing a border as a file transfer, or the competitive pressure between states that shapes how fast everyone builds.

The treaty layer

That gap points to the third candidate: international coordination. The 2023 summit at Bletchley Park produced the first joint statement on frontier AI risk signed by the United States, China, the European Union, and two dozen other governments. Successor summits followed, and several countries stood up AI safety institutes — small public bodies staffed to actually test frontier models. Modest, but genuinely new: technical evaluation capacity inside government, where almost none existed before.

Skepticism is warranted. Declarations are not treaties, nothing signed so far binds anyone, and the summit agenda has drifted from safety toward investment and national advantage. Still, one feature makes AI more governable internationally than software ever was: physical chokepoints.

Training a frontier model requires vast clusters of specialized chips, produced by a supply chain with only a handful of firms at each critical stage. Compute is countable, trackable, and located somewhere — none of which is true of code. The better analogy is not filesharing, which no treaty ever contained, but fissile material, which treaties partially did. This is why chip export controls exist, and why verifying a future international agreement is conceivable at all. The window may narrow as training becomes more efficient, but for now compute is the strongest handle any outside governor holds.

The confounder: open weights

Any honest framework has to absorb an awkward fact. Some highly capable models are released with open weights — the trained parameters published for anyone to download. Once that happens, the weights are copied across the world within days. Built-in safety behavior can be stripped out by anyone with modest resources and a fine-tuning script. There is no recall.

This is not an argument for or against open release. Open weights bring real benefits — independent scrutiny, competition, access for researchers and poorer countries — alongside real irreversibility. The point is narrower: openness splits the governance problem in two. For closed frontier models, you can govern development — what gets built, tested, and released. For open models, only governance of use remains, the way societies govern chemistry knowledge rather than chemistry itself. Any regime that pretends otherwise is decoration.

The pacing problem

There's a structural mismatch underneath all of this. The EU began drafting its AI Act before ChatGPT existed and had to bolt on general-purpose model rules midway through passage. Capability cycles now run in months; legislative cycles run in years, followed by more years of implementation. Any law that names specific techniques is describing where the technology was, not where it is.

The standard fix is to write outcomes into statute and delegate technical detail to agencies and standards bodies that can update faster. That helps, though it shifts power toward institutions with thinner democratic legitimacy and, often, thinner expertise. The deeper fix is capacity: a government that can independently measure what a model can do doesn't have to legislate blind, and doesn't have to take a lab's word for it.

Thresholds and uses

So who should govern AI? If the question expects a single name, it's the wrong shape. A more useful frame is to notice that rules can attach at two different points.

Capability-threshold rules attach to the model itself: if a system crosses defined lines — scale of training, evaluated abilities in dangerous domains — obligations trigger, whatever the intended use. Testing, disclosure, security requirements. Lab safety policies are this idea in voluntary form; making it law targets the small number of frontier developers and gives the public leverage over what gets created in the first place.

Use-case rules attach to deployment: hiring, medicine, weapons, children. The application is regulated no matter which model sits underneath. The bulk of the EU Act works this way, and crucially, it is the only kind of rule that still touches open-weight models. It governs harm where harm actually lands.

Each layer fails alone. Threshold rules can't reach the million mundane deployments where most damage occurs, and they can't reach open weights at all. Use-case rules can't see the next capability jump coming until it has arrived. A workable regime almost certainly needs both — plus a third, unglamorous ingredient: real measurement capacity in the public sector, meaning evaluators, incident reporting, and technical talent that doesn't depend on industry goodwill.

You cannot govern what you cannot measure — and right now, almost all the measuring is done by the governed.

The honest conclusion is that "who should govern AI" remains open. The labs govern by default, nations govern in part, treaties barely govern at all, and open weights limit what governing can even mean. That openness is not a reason to look away. It is precisely why the question deserves sustained attention: the answer is being written now, one shipped model at a time, and it will be written whether or not anyone is deliberate about it.