All walkthroughs · Paper briefs

The AGI Race and Existential Risk

Ethan Bueno de Mesquita, Wioletta Dziuda, Mattias Polborn
NBER Working Paper 35276, May 2026 (also BFI Working Paper 2026-71)

Paper · Brief and rating · Dashboard entry

About 21 minutes · Download MP3

AI-generated walkthrough with synthesised voices: a host explains the paper and a discussant questions it. It covers claims, methods, key assumptions, citations and use so far, decision relevance, and what an evaluator should check. Every direct quotation was checked word for word against its source. Public sources only. Not an Unjournal evaluation or decision.

Transcript

Host
This is a walkthrough of The AGI Race and Existential Risk, by Ethan Bueno de Mesquita and Wioletta Dziuda of the University of Chicago and Mattias Polborn of Vanderbilt. It is National Bureau of Economic Research working paper three five two seven six, released in May twenty twenty-six. AGI stands for artificial general intelligence, meaning systems that can do a wide range of valuable cognitive work at or above human level. This walkthrough is AI-generated. It is not an Unjournal evaluation and not an Unjournal decision about the paper.
Discussant
Start with the question. There is already a literature on AI races where firms skimp on safety to win. What is new here?
Host
The margin they model. Each firm has a fixed budget, think of it as compute, engineers, energy or managerial attention, and splits it between speed and caution. The abstract puts the trade-off directly. The authors write: "Speed increases a firm's chance of reaching AGI first but leaves fewer resources for safety; safety lowers doom risk but slows arrival." Doom is their word, borrowed from frontier AI discussions, for a catastrophic outcome at arrival, whether from a misaligned system, premature deployment or misuse. Because speed and safety draw on the same budget, they can ask how the number of firms, the size of each firm's budget, and the distribution of a fixed industry pool across firms change the split. That lets them compare consolidation, compute policy, public entry and commitment devices in one framework.
Discussant
And the headline results?
Host
Four. First, more firms means more total speed and a higher probability of doom conditional on arrival. Second, there is a critical number of firms, which they call N sub c. Below it, each firm's expected payoff from AGI arriving is positive. Above it, the expected payoff is negative, yet firms keep racing. In their words: "The race can therefore become privately self-defeating: firms may continue to race even though, conditional on an arrival occurring, each firm's expected payoff is negative." Third, giving each firm more resources always brings AGI forward, but it makes arrival safer below the cutoff and riskier above it. They write: "Thus resource abundance is not mechanically good or bad for safety; its effect depends on market structure." Fourth, a set of policy results: a planner who can only choose the number of firms picks a monopoly, regulation can work as a commitment device that firms themselves value, and a cautious public entrant can raise welfare.
Discussant
Let's get into the model, because those results will live or die on functional forms.
Host
There are N identical firms, each with resources R. Firm i picks speed s and caution c with s plus c equal to R. The race then runs in continuous time. Each firm has a random threshold drawn independently from an exponential distribution, and it reaches AGI when its speed times elapsed time crosses that threshold. So each firm's arrival time is exponential with a rate proportional to its speed. The first firm to arrive wins. With some probability the winner's arrival ends in doom, in which case every firm loses an amount D. Otherwise the winner gets a prize B and everyone else gets zero. Payoffs are discounted at rate rho.
Discussant
And the doom probability?
Host
It is an exponential function of the winner's caution per unit of speed. Specifically, the probability of doom equals e raised to minus alpha times the quantity caution divided by speed. Since caution is R minus s, that is the same as minus alpha times R over s minus one. They call R over s the safety ratio. Alpha is intrinsic safety: a higher alpha means a given amount of caution per unit of progress buys more safety. The exponential race gives two useful properties: the probability that firm i wins is its share of total speed, which is a standard Tullock contest success function, and the identity of the winner is independent of when arrival happens.
Discussant
So the safety technology is scale-free. Doubling both speed and caution leaves doom risk unchanged. That will matter later. What about the equilibrium?
Host
They show there is a unique symmetric Nash equilibrium with interior speed. The first-order condition has four terms, and the paper uses them to tell the story. There is a timing term: speeding up brings arrival forward, which is good if arrival has positive expected value for the firm and bad if negative. There is business stealing: more speed takes win probability from rivals. There is what they call doom stealing: if rivals are less safe than you, speeding up shifts arrival away from dangerous rivals. At a symmetric profile all projects are equally dangerous, so this term is zero. And there is the doom penalty: more speed lowers your own safety ratio, which raises the chance that your own arrival is a catastrophe.
Discussant
Why does individual speed rise with the number of firms? In standard contest models, effort per contestant usually falls as the field gets crowded.
Host
They address that directly. The authors write: "That standard dilution force is present: a marginal increase in speed has a smaller effect on a firm's probability of winning as the race becomes more crowded. But the internalized doom penalty is also diluted." The firm only bears its own doom penalty in the states where it is the winner, which happens with probability one over N. Business stealing depends on the marginal effect of speed on win probability, which at a symmetric profile is proportional to N minus one over N squared. The ratio of the two is N minus one over N, which rises toward one as N grows. So holding the safety ratio fixed, more firms raise the relative value of speed, and the equilibrium safety ratio has to fall.
Discussant
So the result rests on doom being something every firm suffers equally, while each firm counts the extra risk from its own speed only when it is the winner. It is an externality argument with a contest attached.
Host
Yes. And the critical number of firms comes from the same structure. The expected payoff at arrival is one over N times B times the success probability, minus D times the doom probability. That falls with N for two reasons: each firm is less likely to win, and the race is riskier. The cutoff depends on B, D and alpha, but not on R. The reason is the scale-free safety technology. At the break-even speed the timing term drops out, and the other two terms depend on speed only through the safety ratio, so changing R just rescales speed.
Discussant
Why would firms stay in a race with negative expected value? They could leave.
Host
Leaving does not protect them. Doom caused by others hits them anyway. In an appendix they let firms choose between entering and an outside option while staying exposed to doom from entrants. The relevant cost of entry is only the incremental doom exposure, since a firm that stays out bears the rest anyway. They show numerically that equilibrium entry can land on either side of the cutoff. In their illustration, with prize and doom loss both set to one, alpha one half, and discount rate one tenth, the cutoff lies between five and six firms, and different outside options give three entrants or seven.
Discussant
I would flag that example. Between five and six firms sounds like the actual frontier, and a listener will be tempted to read it as a calibration. It is not. The parameter values are arbitrary normalizations.
Host
Agreed. The paper does not claim a calibration. Moving on to resources. With a fixed number of firms, a larger budget per firm always raises speed. The effect on safety runs through the timing term. If arrival is valuable, the extra benefit of bringing it forward shrinks as the race gets faster, so firms put relatively more of the new resources into caution, and the safety ratio rises. If arrival has negative value, the restraint coming from the timing term weakens as the race speeds up, so firms put relatively more into speed and doom risk rises.
Discussant
And when the industry total is fixed and the number of firms changes?
Host
That is their fragmentation result. Split a fixed pool among more firms, and total speed rises, the share of the budget spent on speed rises, and doom risk rises. Read in reverse, consolidation is safety-enhancing through this channel. They are careful to say that concentration raises market power, accountability and misuse concerns outside the model.
Discussant
The authors are explicit about that. The authors write: "The analysis should not be read as a general argument for concentration, resource restriction, or public provision." Now the policy section.
Host
The planner puts weight K on doom losses, at least N, so possibly far larger to capture harm to everyone outside the firms. With full control, the planner equalizes safety ratios across projects and then does not care how many firms there are. It picks lower aggregate speed than a monopolist would with the same total resources. The paper summarizes the restricted case: "A planner who controls only the number of firms prefers a monopoly." If the planner can only set total industry resources, the answer is bang-bang. If the number of firms is above the cutoff, shut the sector down. If it is below a related cutoff that shrinks as the planner's weight on doom rises, give the industry the maximum. So the second-best can combine consolidation with resource subsidies.
Discussant
Then the commitment result.
Host
Suppose one firm can commit to a speed before the others choose. Committing to slightly less than the equilibrium speed raises its payoff, because rivals respond by slowing down. The mechanism is the doom-stealing term: if the leader is safer, losing to it is less dangerous, so rivals have less reason to race. The authors write: "Commitment to caution is privately valuable because it induces caution by others." A corollary says a small commitment of this kind makes all firms better off. They connect this to Anthropic's original Responsible Scaling Policy, which committed to pause scaling if safety procedures lagged, and read its later move away from a categorical pause as the commitment problem in practice. They also use it to interpret industry calls for regulation: a firm may want common rules because they change rivals' incentives.
Discussant
And public entry?
Host
With at least three private firms, a public firm that moves slowly enough raises welfare. There is a direct effect: sometimes the safe public project wins. There is a strategic effect that changes sign. When the private race is above the cutoff, the public entrant makes private firms slow down. Below the cutoff it makes them speed up, because it adds competition. The result holds because the direct effect dominates when the public project is cautious enough. The authors write: "Public entry is therefore useful only if it is structured as a cautious alternative, not as another aggressive racer." They also say the result ignores the fiscal cost of the public firm's resources.
Discussant
Let me press on robustness before we talk about uptake. Which assumptions carry the argument?
Host
Four do most of the work. Winner-take-all with a single arrival ending the game; doom determined only by the winner's own safety ratio; the scale-free form of that doom function; and symmetric firms with a fixed budget chosen once. On winner-take-all, they show that giving losers some payoff b leaves results intact as long as b is small relative to B. On distributions, they say much of the analysis extends to proportional-hazards and Weibull distributions, though in a footnote rather than a proof. On fixed resources, they argue it is a short-run constraint. The authors write: "This fixed-resource assumption is meant to capture a short-run constraint, not to suggest that the frontier-AI sector cannot expand over time." On exponentials generally: "The tractability of the exponential allows us to keep the analysis focused on substantive, strategic forces rather than distributional details."
Discussant
Here are my concerns. One, safety work is partly a public good. Interpretability tools, evaluation methods and alignment techniques spill across labs. In this model a firm's caution protects only its own arrival. If caution spills over, consolidation's advantage could shrink, and fragmentation might even raise total safety spending. Two, the cutoff being independent of R is a product of the scale-free doom function. If safety has fixed costs or returns to scale, the sign switch for resource effects would move with budgets, and the clean diagnostic goes away. Three, symmetry. Real labs are unequal, and the doom-stealing term only comes alive with asymmetry. A recent experiment found that people who fall behind in a stylized race choose unsafe options more often. A leader-laggard version of this model could look quite different.
Host
The authors list heterogeneous firms among their own limitations. In the conclusion they write: "It abstracts from heterogeneous firms, endogenous investment in resources, uncertainty about safety technology, multi-stage development, and the opportunity cost of public resources."
Discussant
Four, the static choice. Firms choose once. The memoryless exponential means they would not revise even if allowed, which the paper presents as a feature. But real labs observe rivals' progress and adjust, and that is where racing dynamics probably bite hardest. Five, international competition. They exclude geopolitics. Domestic consolidation does nothing to the count of firms if the relevant race includes Chinese labs, and the policy conclusions about monopoly and subsidies could flip. Six, the policy diagnostic. The planner's key question is whether arrival is good or bad in expectation. That depends on B, D, alpha and the planner's weight on doom, none of which anyone can measure with confidence. The bang-bang result means a small error in that sign moves policy from maximum subsidy to shutting the sector down.
Host
On disclosure, the paper notes that Bueno de Mesquita has advised Meta on governance issues. That is stated on the title page.
Discussant
Now uptake. What has actually happened since May?
Host
Mostly academic and institutional communication so far. The Harris School and UChicago News ran a story in late June twenty twenty-six. In it, Dziuda is quoted: "The more firms you have, the riskier the race becomes as the firms try to outpace one another." A trade outlet, The AI Insider, covered it in early July. The Becker Friedman Institute published a research brief on September thirtieth, twenty twenty-six, based on the institute's working paper version. It has also been on the seminar circuit, including Illinois in April.
Discussant
Citations?
Host
Semantic Scholar shows two citing works. One is a behavioral experiment by Fernández Domingos and Han on an idealized AI race, which cites it alongside other theory models where competition pushes toward riskier development. The other is a paper by Burtsev on recursive AI self-improvement, which cites it as one source on how competition shapes research effort. Both are literature-review citations. OpenAlex lists no citing works yet for any of the three records, the NBER version and two SSRN versions. The Altmetric score recorded on the briefs page is thirteen, from one news outlet and a handful of social posts. I found no policy document citing it. A search miss leaves this uncertain, and the paper is only five months old.
Discussant
One thing an evaluator should look at is how the brief simplifies the results. The institute's summary says: "Industry consolidation, regulation facilitating commitment to slow down, and governments entering the race in a safe way improves welfare, while restricting resources may backfire." In the paper, resource restriction is the optimal second-best policy when the number of firms exceeds the cutoff. Whether it backfires depends on which side of the cutoff you are on, so the one-line summary leans one way.
Host
The press story makes a similar move, saying that with fewer players more resources may improve safety. That is accurate, but readers could take it as an argument for subsidizing incumbents.
Discussant
Why might this matter for decisions?
Host
Three audiences. Regulators and competition authorities deciding on mergers, partnerships and exclusive compute deals among frontier labs: the paper gives a formal channel by which consolidation lowers catastrophe risk, to be weighed against antitrust harms. Policymakers on compute: export controls, reporting thresholds and public compute programs like the National AI Research Resource, which the paper mentions, change different objects in the model. And funders considering public or nonprofit AI projects: the result that a public entrant helps only if it is slow is a sharp design condition. Labs may cite the commitment result when arguing for binding common rules.
Discussant
And why would it suit an Unjournal evaluation?
Host
It is a clean theory paper by established political economists, it speaks directly to live policy arguments, and the press framing already goes a step beyond the propositions. There is no data to replicate. The evaluation is about whether the proofs hold, whether the key results survive reasonable changes to the assumptions, and whether the policy readings follow. A theorist in industrial organization or contests and a reader who knows how frontier labs actually allocate compute would complement each other.
Discussant
Then here is a checklist for that evaluator. Verify the uniqueness proof and the comparative statics, especially the claim that individual speed rises with N. Test how much depends on the scale-free doom function by trying a version with fixed safety costs. Add safety spillovers across firms and see whether consolidation still lowers risk. Try asymmetric budgets and check whether the doom-stealing term reverses any results. Look at whether the commitment result survives when the commitment is non-local or imperfectly credible. Ask whether the footnote claims about Weibull and proportional hazards distributions are proved anywhere. Assess whether the critical-number-of-firms concept has any empirical handle, or whether it is a qualitative warning. And check how the public summaries describe the resource results against Propositions four and eight.
Host
In short, a tractable model in which market structure changes how firms split scarce inputs between speed and safety. Its policy conclusions are conditional, and the conditions are hard to observe.

Sources