In his much-vaunted, five-volume LBJ biography, Robert Caro deploys a number of striking phrases—often direct quotes—and uses their repetition to drive home a particular point. In The Passage to Power there is one excellent phrase that Caro uses to make a dual point about the nature of political power in the mid-20th century and LBJ’s particular facility in using it. LBJ does not want to contest the 1960 Democratic nomination. He is hoping instead that the convention will be deadlocked, and that then, freed from quasi-democracy, he will be able to achieve his aims in the ‘smoke-filled rooms’ where real power lay, as he had done before many times in politics and business both.
To their credit, the leaders of AI companies, their critics and their regulators seem unusually disinterested in the smoke-filled room. They prefer the roon-filled squares of our public forum, the internet. Long may it reign.
The Last Company On Earth
You may have seen, for example, the rather lively exchange between Gavin Baker of Atreides Management and Sholto and Dario of AI lab Anthropic. Their exchange is too long to usefully summarize here in its entirety, but there were many questions raised that have a much wider bearing.
One of these is the loaded question of “regulation”. Anthropic, Baker says, would like to be the only company on earth. Failing that, it at least holds a view of the world in which “AI is too dangerous to distribute”, whereas he and others feel that “AI is too dangerous to centralize”.
This is a little black and white for my taste—the US has decided, for example, that firearms are both too dangerous to centralize and too dangerous to distribute. There are entire worlds of policy discussion that could fit in the difference between two degrees of TDTC and TDTD. But it is also a) an accurate description of the views of some people in the world of AI safety, and so not totally without precedent, and b) not entirely outside the overton window, as far as the shape of future policy debates goes.
Dario himself makes a version of this case:
But in particular I think that those in the “regulation = regulatory capture = concentration of power” frame often underrate the decentralizing power of objective and fair institutional processes.
And, if you’ll allow me to be glib for a moment, I think you could characterize the views of almost every serious leader in the AI space (including in the Trump administration, other labs, the hyperscalers, the organs of the Chinese Communist Party etc etc.) as occupying some of this nuance. Mark Zuckerberg (who Baker quotes approvingly) is not just calling for a role for civil institutions from across society in ensuring a balance of power for AI, he also specifically calls for safeguards on biorisk and cyber-risk, and preparedness for if things start to go really wrong.
OpenAI is pausing. Google has been pausing. Demis is writing up entire regulatory plans himself. GLM-5.3 is doing its own Glasswing cyber-risk slow roll. The DoW has very strong views on how leading AI models ought to be regulated and it’s willing to wield direct political power to ensure they are. Andy Jassy said the Fable safeguards were too soft. Both ends of the political horseshoe are losing their minds about data centers. Elon and SpaceXAI think that, at the very least, we should think hard about safety and alignment.
State Incapacity
There is a sleight-of-hand I am on the verge of performing here. Just because I can list a group of people who could be verballed into supporting regulation in some narrow, specific sense, does not mean they are Against Gavin Baker and In Favor of Regulation. Each of these people would likely be able to point to many very bad pieces of regulation that they would like to remove or change.
The regulatory state has not, as a rule, covered itself in glory during the 30 years since they invented the internet. There are the many failures of financial regulation. There is, of course, the morass of healthcare. Don’t get me started on the universities. I’m confining myself to the US because talking about Europe, or even Australia, will make me sounds like one of those cranks that paints anti-state messages on my busted Camry in acrylic children’s paint.
Gavin and others are most concerned, in the case of AI, about the idea of regulatory capture. It has been a goddamn rodeo out there folks. Don’t click on those links, they’re just for show. Sometimes an industry captures a regulator so hard, as in the case of the New York taxi system, that it amounts to a kind of heritable peerage.
How confident are we that the state is going to get this right? If you believe that AI is the defining technological wave of our time, the stakes are enormously high. And there are precious few examples of success to draw on.
There is the risk that Baker is concerned about, of course. Regulatory capture. A frontier AI regime so strict that it prevents all but the incumbents from bringing models to market and competing with the existing frontier labs. Perhaps a hereditary Member of the Technical Staff title, that you can pass down to your children or favored pet.
But there are also almost unimaginable numbers of other failure modes. There is the regulation that is both ineffective at promoting safety, and conducive to capture. There is the regulation that seeks to redistribute the gains of AI based on the economic structures—firms, states, employees, trades, certificate-holders of all kinds, clinical trials—in place at the moment the regulation is passed, foregoing the enormous benefits that could be created by rearranging those structures in light of new technology.
No one mention the dock workers.
The AGI State
The problems in these cases are not just matters of centralization or decentralization. They are, above all else, an expression of the opposite of state capacity. State incapacity (or, in some cases, a kind of active state malevolence).
(I don’t want to overstate my case—many things do work. We live in a time of historic plenty. When you press the switch, the lights magically come on. Sometimes regulation is the result of an ugly compromise, and that’s just how it works, etc, etc.)
But the alternative to the problems of “regulation” or state action in the broadest sense is not necessarily a world of perfect competition and competitive markets.
This is a world in which there is no—or perhaps only limited—regulation in the sense that Gavin Baker talks about. But the same decisions are being made. They are being made individually by the labs, in an environment of relative uncertainty about how the population, how the demos, how the state might react to a failure of self-regulation by the frontier labs.
In some ways, the alternative to state action is what we have now. Corporate action. The decisions—commercial, policy and regulatory—that companies make either alone or in cooperation with each other.
Frontier companies in the US, those that are close behind or at par on capabilities like xAI and Meta, the US open ecosystem comprised of Thinking Machines and hyperscalers and enterprises, and of course, the Chinese open-source ecosystem—all operating in an ecosystem in which they face short-term and long-term interests, uncertainty, asymmetric information, principal-agent problems and so forth.
So it leaves us, I think, with a question far more complicated than simply: do we distribute AI, or do we concentrate AI?
The labs and hardware companies and hyperscalers and enterprises are filled with moral and principled people who have their own views and their own ideologies about how to make AI go well. But they are also full of clever, strategic decision-makers who are trying to factor into their decisions not just what they are “allowed” to do, but what the consequences of their actions will be for a volatile public.
So it may well be that the decision by the state not to regulate leads to more conservatism, to a slower and more cautious approach by the leading labs. And, of course, it may go the other way entirely.
This ecosystem is not at all a naive, profit-maximizing ecosystem. It has far more baroque and complicated decision-making frameworks than simple cynicism might suggest.
Balaji has long been developing the concept of the Network State. Kazakhstan looks glorious this time of year. It is hard for me to avoid the conclusion that there might be—centralized or decentralized—many possible AGI States developing. “Regulation”, in the general sense, is no different to the Zuckerberg quote that Baker likes: an ecosystem of checks and balances, managing the many competing and overlapping interests of these AGI States with those of the states and groups that preceded them.
And there is, for better and worse, no way of doing that without staring down the bullet, one SB 53 at a time, and deciding whether to bite it, or not.
.jpg){kind=link}