Skin in the Game on AI Risk

Rapid, self-improving AI, our growing inability to understand and control these models, rogue agents, and incidents such as OpenAI-Hugging Face (OAI-HF) have set off alarms over the need to safeguard and pace the frontier.

At the head are Anthropic, OpenAI and other frontier labs. However, risk travels and compounds all along the AI chain. Downstream providers build on these models, orchestrate them into increasingly autonomous systems (agents), and sell them to enterprises around the world with access to our data, bank accounts, and infrastructure.

We believe the solution to AI risk will hinge on speed, liability, and alignment. Rather than pacing AI, we believe in pushing it forward with real skin in the game, built on the same accountability structures that enabled medicine, aviation and nuclear energy.

The incentive problem

There has been a call for democratic coordination, common standards, and third-party evaluators of AI risk. The problem is:

1. This is slow. AI risk evolves rapidly and continuously while laws take time. Even if regulation could catch up, market forces push against this, as pacing too much would hand the AI lead to China or less scrupulous players.

2. There is no clear liability. The elephant in the room in AI risk. Who pays when AI goes wrong? Without regulation or liability, labs and AI companies have no strong incentive to seriously and proactively safeguard AI risk.

Dario Amodei proposes closing this gap by involving independent, third-party evaluators, or watchdogs of AI, verifying safety practices and auditing models for dangerous capabilities before release. However, this raises a third problem:

3. Can we trust the evaluator?

Anthropic is funding its own evaluator, Accenture, whose shares jumped 8% after the deal was announced. Independent evaluators are hard to come by.

The incentives also discourage rigor. The space of possible failure and edge cases in AI systems is massive. While catching the first n failures is cheap, catching n+1, n+2, and beyond becomes disproportionately difficult and expensive. An evaluator bearing none of the cost of failure is rarely incentivized to search beyond n, and attempts to enforce rigor through regulation have devolved into similar box-ticking exercises, limited to fixed, pass/fail tests rather than proactive, thorough assessments of evolving AI risk.

Enterprises face the same problem with eval, red teaming, and security vendors testing their agents. If an agent leaks data or wires funds to the wrong account, the vendor is almost never responsible for the cost. Often the failure is never even traced back to the evaluation, because the failure surfaces late, if at all, and is hard to attribute. Evaluators who bear no cost for an inadequate assessment inevitably tend toward procedural, weakly incentivized work.

A market-based mechanism for AI safety

This is not the first time we face runaway technological risk. A week after Dario Amodei’s “We Must Pace the Frontier” essay, Yoshua Bengio urged the UN Security Council to treat AI like medicine, aviation, and nuclear energy, underwritten and secured via liability insurance. In other words, to put skin in the game on AI risk.

This solves the three problems above.

1. Speed. Unlike regulation, which is slow, insurance reprices continuously at the pace of risk. When new capabilities emerge (self-improving models, autonomous personal agents…) and incidents occur (e.g. OAI-HF), insurers reassess their exposure and price it into premiums across the AI supply chain (from model to agent to enterprise). Safer labs, agent providers, and enterprises pay less, turning safety into a competitive advantage. The frontier innovates at the speed of the market, with the market now attuned and reactive to risk.

2. Liability. Insurance defines (in advance and in writing) who is liable, which failures are covered, how much is paid out, and to whom. After an agent leaks data or wires funds to the wrong account, the enterprise is made whole. The insurer, which can pursue the responsible party upstream, has every reason to trace the failure back to where it began. The fault may lie in the model (AI lab), agent (downstream provider), or deployment (enterprise itself). Liability no longer evaporates throughout the chain and instead lands wherever the risk originated. Exposed to the cost of incidents, there is suddenly a real, economic motive to better understand, test, and evaluate AI systems before release.

3. Alignment. Insurance is an $8 trillion incentive system. Historically, from cars to electricity, it has allowed technologies to scale safely by aligning risk prevention with the market. Beyond assigning liability, insuring a new risk mobilizes the entire industry to:

i) Evaluate the risk deeply (because it must predict losses and price them)
ii) Prevent that risk from happening (because it will pay the claims)

In a liquid market, any bias, error, or conflict of interest in achieving (i) and (ii) is corrected by competition. If an insurer overestimates AI risk, it will lose to competitors. If it underestimates it, it will pay it in claims. This creates a powerful incentive to get ahead of AI risk as capabilities advance. Historically, the insurance industry has partnered with technical experts in achieving this task, using its massive financial weight to accelerate safety research, industry standards, and rigorous, highly adaptive risk evaluation.

The insurability of AI

Insuring AI comes with its challenges. AI is new, dynamic, opaque, and highly concentrated. Creating this new category will require highly technical, continuous underwriting, clear liability along the chain (model to agent to enterprise), and pooled capital for correlated risk. None of this exists at scale yet, but the pieces are forming. The next article, “The Insurability of AI,” will take each of these challenges in turn and describe what an insurance market built for AI should look like. More to come.