Who Gets to Define the Rules for AI?

A perspective from Aidan Gomez, Co-founder & CEO of Cohere Artificial intelligence is remaking the world we live in. Within a generation, the way we discover medicine, manage power grids, and secure our national infrastructure will be completely transformed. Many people already realize this and are working to build that future responsibly. Many people underestimate the scale and pace of change coming. Some, however, claim to foresee this change and use it to serve their own ends. Here is the question nobody is asking clearly enough: should a handful of select, market-dominant AI companies from Silicon Valley get to define the rules and safety standards of a generational technology for the entire world? All while simultaneously determining how fast this technology progresses? We have tried that before with very poor results. Once again using fear under the pretext of protecting the public, these oligopolies are now requesting to bend competition rules and be permitted to dictate the terms for everyone else. A wolf in sheep’s clothing, a cartel by any other name. I believe in the potential of AI technologies to bring benefits to our world, and I do not downplay the risks. I run a company that builds AI systems deployed inside banks, telecommunications networks, and defense ministries. These are among the most high-stakes environments because failures in these sectors can have consequences far beyond an individual user - disrupting financial systems, critical infrastructure, national security, and essential services at scale. The same capabilities that find vulnerabilities in your code can find them in someone else's, and cyber offense is getting cheaper faster than defenses are getting better. That gap should worry you as much as it worries us. AI needs guardrails. That is not the dispute and never has been. The dispute is over who writes them, who gets to participate and whose interests the rules are protecting. The question is truly about whether we should have the freedom to choose based on scientific evidence or if we should hand the reins of the most consequential technology in human existence to a few Silicon Valley executives. We’ve been here before Before I break down the self-serving framework the big labs are pushing and offer ideas for an alternative, let’s take a couple lessons from the recent past and think about the word cartel, because the history is specific and it is the accurate term. In 1975 the Securities and Exchange Commission needed reliable bond ratings for its capital rules. It designated three firms as Nationally Recognized Statistical Rating Organizations and never published criteria for how anyone else might earn the designation. These were government-blessed outside evaluators, paid by the very issuers whose securities they graded, sitting behind a barrier the regulator itself had built. Twenty five years later there were still only three of these evaluators. Then they rated subprime mortgage securities triple-A and nearly took the global economy down with them. Europe ran the experiment again in 1985. Car manufacturers lobbied for a sweeping antitrust waiver, the Motor Vehicle Block Exemption, arguing that modern vehicles were complex, safety-critical machines and that manufacturers therefore needed control over who was qualified to sell and service them. The subsequent regulation let manufacturers set the standards for premises, equipment and staff training, explicitly in the interest of safe and reliable vehicles. But what followed wasn't safer cars. It took the European Commission roughly twenty five years of reforms to unwind, and to establish what should have been obvious at the start: it is possible you can hold strict safety standards without handing the incumbents a monopoly on meeting them. Nobody set out to build a cartel in either case. In both cases, the stated goal was safety. But the result was a market structure that protected incumbents and limited competition, all under the justification of serving the public interest. I don't doubt the sincerity of the people involved: many were concerned about the risks and worked earnestly to resolve them. But complex problems aren’t always solved on the first try, and any responsible scientist, engineer, or lawmaker knows that to solve new problems you must learn from past history. What's Being Proposed This brings us to the roadmap published this week by Anthropic CEO Dario Amodei, asking governments for antitrust exemptions in the name of safety. This roadmap is the latest in a string of recent efforts by Silicon Valley incumbents to shape the regulatory landscape surrounding AI. I want to be clear about what we agree with. Independent review of highly capable AI systems is a good idea and we support it. However, many aspects of the proposal raise fundamental questions: who writes the standard those reviewers apply? Who conducts or oversees the review? Who gets to participate in the conversation that sets the rules? On these questions, the proposal is clear. A handful of the most powerful labs based in one country would agree on shared standards and the limits to how fast the technology should advance. And here’s the key point: because it’s normally illegal for competitors to agree to limit what they produce, the plan asks governments for a narrow antitrust waiver to make that coordination lawful. And it also asks governments to require every other AI developer to blindly follow whatever the participants settle on - despite those other developers and wider society not having an opportunity to voice the impact or share their perspective on the science. This is not a question of adding one or two more companies into the conversation. Adding an extra chair fundamentally doesn’t solve the issue. The problem is that there is a list at all, when the decisions being made reach every company, every government, and every citizen who never got asked. You cannot have it both ways. If this is the most consequential technology in human history, then the rules for it cannot be written by a small group of commercially aligned companies behind an antitrust waiver. There is no public comment period here. There is no consultation, and there is no vote. The public will be forced to live with the outcome regardless. A safety regime designed by a few labs will only be rigorous about the risks they have already built their safety systems to assess and completely quiet about everything else, further entrenching their market position and limiting competition. Risk in these existing frameworks gets defined as a function of scale, which makes the companies with enormous systems the only ones qualified to judge. The types of risk deemed relevant for assessment are also pre-ordained, rather than up for scientific debate and alignment. For example, there is real disagreement in the field about how much offensive capability comes from a raw model size versus the harness wrapped around it. Smaller models orchestrated well, using tools and verification steps, can do things that large models can’t. A cyber swarm is a completely different risk surface than a single model. None of that shows up in a regime built exclusively around massive compute thresholds. There's a sentence in the essay that any competition authority would find troubling. It promises that a coordinated approach would give developers time to do this safety work without sacrificing commercial advantage. But to whose advantage? The firms drafting the framework are the firms sitting at the top of the market today. A mechanism that slows everyone down while explicitly preserving existing commercial advantage does not make AI safer. It risks entrenching today’s dominant AI companies by turning their current advantages into baseline for what it takes to compete safely. Safety rules should reduce risk without regard to who leads the market or who stands to gain from the rules. The entry requirements set out in the proposal tell you the rest. Vast computing power. Continuous monitoring infrastructure. Dedicated security organizations. Resident evaluator teams with desks and badges. Shutdown architecture. Government relationships that are deep enough to navigate all of it. A pool of “independent” evaluators that is already remarkably small, funded by the same handful of organizations repeatedly relied upon by the same frontier labs. Convince a government that AI is an existential threat and you can convince it to outlaw your competition. The intention is clear and it does not create a safer world. What Better Rules Look Like So what will enable safe, responsible AI development? To be clear, I don’t believe I have all the answers - nor do I think I should get to make the rules instead. Rather, I will try to propose practical and effective ideas that can be considered alongside those of many others by governments and lawmakers as they use their democratic powers to set the direction of travel. Those ideas are built on four pillars: - An evidence-based risk framework. First things first, and before anyone mandates testing or auditing, we need an agreed and published account of which harms we are concerned about, which AI capabilities cause which harms, under what conditions and in what contexts, and at what point a government should step in. That account must be built across all the countries developing this technology, and in the open rather than behind closed doors under the banner of national security. Establish a coordinated, international effort to develop this framework that is not led by any one nation, but a group of them. Put technologists in the room next to the policy experts and experts from critical sectors like finance and critical infrastructure. Include researchers and scientists who disagree with each other and publish the disagreements, because an honest process shows its arguments instead of announcing its conclusions. Fund the testing capacity itself through public research bodies an