All walkthroughs · Paper briefs

Cooperating against Catastrophe

Drew Fudenberg (MIT) and Andrew Koh (Columbia)
arXiv 2609.28291 [econ.TH], version 1, 23 September 2026 (title page dated 24 September 2026)

Paper · Brief and rating · Dashboard entry

About 21 minutes · Download MP3

AI-generated walkthrough with synthesised voices: a host explains the paper and a discussant questions it. It covers claims, methods, key assumptions, citations and use so far, decision relevance, and what an evaluator should check. Every direct quotation was checked word for word against its source. Public sources only. Not an Unjournal evaluation or decision.

Transcript

Host
This is a walkthrough of Cooperating against Catastrophe, by Drew Fudenberg of MIT and Andrew Koh of Columbia, posted to the arXiv preprint server in the economic theory section on the twenty-third of September, twenty twenty-six. It is an AI-generated walkthrough with a host and a discussant. It is not an Unjournal evaluation and not an Unjournal decision about the paper. The paper is a dynamic game between two AI developers. Each chooses how fast to push its capabilities, while a safety threshold, which you can think of as the state of alignment research or institutional preparedness, rises at a fixed rate. When the leading firm's capability gets ahead of safety, both firms face a common risk of disaster that ends all profits. The question is posed directly in the introduction. The authors write: "When can self-interested, profit-maximizing firms slow AI research to a safe pace without government intervention?"
Discussant
So this is a theory paper about whether the frontier labs can hold back on their own. Why now? Racing models of AI development have been around for a decade.
Host
The motivation is very recent: Dario Amodei's September twenty twenty-six essay calling for pacing the frontier, and an open statement from more than thirteen hundred employees of frontier AI companies. The authors put the standard obstacle this way: "One obstacle to pacing the frontier is that each firm may reason that if it slows down, a rival will press ahead." And then the counterpoint, citing Schelling: "Yet strategic rivalry can also produce mutual restraint when the parties share an interest in avoiding a bad outcome". Earlier formal AI-race models, including the recent National Bureau of Economic Research paper by Bueno de Mesquita, Dziuda and Polborn on the AGI race, are static. Here firms choose a speed at every instant and respond to each other's progress, in the tradition of the patent-race literature Fudenberg helped build in the nineteen eighties.
Discussant
Give me the main claims before the machinery.
Host
There are four. First, in Markov perfect equilibrium, the parameter space splits into three regions according to a summary of disaster risk they call the effective hazard: low means only racing, intermediate means racing and pacing coexist, high means only pacing. Second, faster safety progress, higher hazard, and more patience push toward pacing, yet very fast safety progress does not eliminate racing. Third, across all subgame perfect equilibria, low risk forces perpetual racing, and disaster probability by any date is bounded. Fourth, if rivals see capabilities only after a deployment lag, pacing is harder to sustain, which they read as an argument for transparency about internal models.
Discussant
Now the model. What exactly do the firms choose and what do they get?
Host
Time is continuous. Each firm picks a development speed between zero, which is pausing, and one, which is racing. Its capability is the integral of past speeds, so capability grows at most one unit per unit of time. The capability frontier is the larger of the two firms' capabilities. The safety threshold rises at a constant rate, call it g, strictly between zero and one. So safety is always slower than a racing firm. The key state variable is the distance of the frontier above safety. If that distance is zero or negative, there is no risk at all. If it is positive, disaster arrives at a hazard rate equal to a possible jump at the boundary plus a function of the distance that is nondecreasing and concave. When disaster arrives, both firms lose all future profits. In the authors' words: "Disaster permanently ends both firms' future profits." Before disaster, each firm's flow profit is the exponential of alpha times its own capability minus beta times its rival's capability, with alpha greater than beta, both positive. So own progress raises profit, the rival's progress lowers it, and equal progress by both raises both. And as they say plainly: "There is no direct cost of development." Firms discount at rate r, assumed larger than alpha minus beta so that payoffs stay finite even under joint racing.
Discussant
Two things stand out. Risk depends only on the frontier's level relative to safety. And there is no cost of going fast except the hazard.
Host
Right on both. The first point is deliberate. They borrow the idea of state risk from Trammell and Aschenbrenner, and they note that "existing capabilities that remain strictly above safety continue to generate risk even during a pause". So once you are above the line, pausing does not make you safe; it only lets safety catch up. The second point means the only reason to slow down is the hazard, plus the strategic effect on the rival. Also note what disaster costs the firms: only their own future profits. In a footnote they say that if firms internalized social losses, "it would be easier to support pacing the frontier and reduce the set of parameters where both firms always race."
Discussant
Continuous-time games are notorious for strategies that do not pin down outcomes. How do they handle that?
Host
With some care. They give an example where both firms race except exactly at the safety line, where they move at speed g; from the same start, both a path that tracks safety and a path that keeps racing are consistent with the strategies, because what you do at a single instant does not change an integral. Their fix is a Krasovskii regularization: at each state, the allowed joint speeds are the closed convex hull of speeds prescribed at nearby states. A profile can then generate several admissible paths, and each firm values a profile by its best admissible path. They prove that against a Markov opponent a Markov best response exists, and they note that other recent work uses a Filippov regularization instead.
Discussant
That evaluation-by-supremum rule looks like a modeling choice that could matter at the boundary, which is exactly where pacing lives. Keep that in mind. What are the two candidate strategy profiles?
Host
Under the racing profile everyone runs at speed one always. Under the pacing profile, a firm strictly behind races, and the leader solves an optimal stopping problem: race to some chosen distance above safety, then pause until safety or the follower catches up. Tied firms race below safety, move at speed g on the safety line, pause if above safety but within a cutoff distance, and race beyond it, where waiting would cost too much in forgone growth and accumulated hazard.
Discussant
And the effective hazard is the summary statistic that sorts the regions?
Host
Yes. Take the normalized payoff from two tied firms racing forever starting on the safety line. The effective hazard is the constant hazard rate that would give that same payoff. It equals the jump term when the hazard is flat above the line and rises when the hazard profile rises. The comparison payoff is perpetual joint pacing from the safety line, which is one over the quantity r minus g times alpha minus beta. Theorem one then says: if the effective hazard is below one minus g, times alpha minus beta, racing is an equilibrium and no Markov equilibrium outcome paces forever from a tie on safety. If the effective hazard lies between that and one minus g, times alpha, both profiles are equilibria. Above one minus g, times alpha, pacing is an equilibrium and racing is not.
Discussant
Give me the intuition for each cutoff.
Host
The lower cutoff comes from a racing guarantee. Start tied on safety. If one firm races forever, it sets the frontier, so it faces exactly the hazard of joint racing, and since the rival can move no faster, its profit is at least the joint racing profit at every date. So any firm can guarantee itself the racing payoff whatever the rival does. If that beats the pacing payoff, pacing cannot survive. One minus g is how fast a racer pulls away from safety, and alpha minus beta is the growth rate of profit under symmetric progress, so the cutoff compares hazard with the growth gain from running ahead of safety. The upper cutoff comes from a different deviation. Against a rival who always races, a leader sitting on the safety line can pace briefly, incurring no hazard, and then resume. Above the upper cutoff that brief safe spell is worth more than the lead it gives up, so universal racing is not an equilibrium. In the middle region, both firms would prefer pacing, but, as the authors put it, "Yet one firm cannot obtain that outcome simply by slowing down: its payoff depends on how its rival responds and what happens when they meet."
Discussant
So in the middle region this is a coordination problem with two equilibria, and the paper does not say which one gets played.
Host
Correct. In their numerical example, a leader who slows down while the follower catches up stays safe under the pacing profile, but against a racing rival the slowdown buys only a temporary safe interval, gives away the lead, and pays slightly less than racing immediately.
Discussant
What about the comparative statics?
Host
Proposition one says faster safety, more risk, and more patience each weakly expand the pacing region and shrink the racing region. Patience matters because a patient firm weights the higher hazards reached later in a race. Under a linear hazard, they show the boundaries converge to strictly positive limits as safety speed approaches one. In their words: "Consequently, safety progress arbitrarily close to the maximum development speed does not eliminate the racing equilibrium."
Discussant
That is a striking policy statement: speeding up alignment research alone does not get you out of the racing trap. Now the subgame perfect results.
Host
Theorem two has two parts. Part (a): if the effective hazard is below the lower cutoff and discounting is strong enough, specifically r plus the effective hazard exceeds alpha squared plus beta squared, divided by alpha minus beta, then every subgame perfect outcome from a safety tie is perpetual joint racing. The argument is that each firm can guarantee the racing payoff, so total payoffs must be at least twice that, and under these conditions every other outcome gives strictly less in total. Part (b): if r exceeds alpha and safety speed exceeds one half, then in every subgame perfect equilibrium starting from a tie on safety, the probability of disaster by any date is bounded above. The bound comes from a recursive payoff guarantee. A firm can unilaterally pause and rely on its rival's own incentive to avoid disaster, which yields a better guaranteed payoff, and that guarantee can be fed back into the same argument at earlier dates. The guarantees converge to a floor, and a floor on payoffs limits how much cumulative hazard any equilibrium can involve.
Discussant
Is the bound informative, or is it a bound that is nearly one in any calibrated case?
Host
The paper states the bound without a numerical illustration that I could find, so an evaluator could reasonably ask for one. Proposition two shows that with a constant hazard, in the coexistence region and when r plus the hazard is less than alpha, a subgame perfect outcome in which firms take turns racing gives both more than the pacing payoff. Proposition three generalizes the profit function: permanent joint pacing is a subgame perfect outcome if and only if pacing beats racing at every safety tie, supported by a grim trigger.
Discussant
Proposition two deserves a pause. The firms beat pacing by taking turns running above the safety line, which is worse for everyone who bears disaster risk without sharing the profits.
Host
Yes. The welfare criterion is the firms' payoffs, so cooperation here means self-enforcing restraint by profit-maximizing firms, which need not be socially optimal.
Discussant
Now the transparency result, which is what the policy summary leans on.
Host
Section three point five adds a fixed deployment lag: a model becomes public, earns profits, and generates risk only after the lag, and rivals see it only then. Using Nash equilibrium, joint racing is always an equilibrium. If the effective hazard is low, no equilibrium sustains joint pacing at any lag. If it is intermediate and the maximum hazard is below one minus g, times alpha, joint pacing is an equilibrium if and only if the lag is no longer than a critical value. If the effective hazard is high, pacing works at every lag. The supporting strategy is a trigger: pace while everything you have seen is consistent with pacing, otherwise race. The authors summarize the mechanism as "the prospect that (belated) discovery that one has deviated might start an AI race disciplines incentives to pace along the safety threshold." And the cost of the lag: "a longer lag allows a firm to build a larger lead before its rival can respond, making a deviation from pacing more attractive." They suggest firms could gain from committing to reveal internal progress before deployment, for example through a third-party embedded evaluator such as METR, which stands for Model Evaluation and Threat Research.
Discussant
Let me push on the mapping to policy. The brief that put this on our list says it supports disclosure of internal, pre-deployment capabilities, of the kind in California's Senate Bill fifty-three, and lab-to-lab or international pacing agreements. How tight is that link?
Host
Looser than a one-line summary suggests. In the model, what matters is that rivals observe internal capabilities. Senate Bill fifty-three is mainly about publishing safety frameworks and reporting incidents; it does not give rivals a real-time view of each other's internal models. And in the model hidden capabilities carry no risk until deployed, a strong simplification given that the paper itself cites a METR report on misalignment risks from internal agent deployments. The relevance to disclosure laws runs through how much they make internal progress visible to rivals.
Discussant
How has the paper been used and cited so far?
Host
It is about two weeks old. OpenAlex, the open bibliographic database, lists zero citing works as of the sixth of October, twenty twenty-six, and the Altmetric score is four, from a few posts on X and Bluesky. A press article dated the thirtieth of September, titled A Way Out of the A.I. Arms Race, which I could only see through search summaries, describes a Fudenberg and Koh working paper proposing that companies and governments share information about their advances and agree on where danger begins. I cannot confirm whether it covers this paper or the authors' July companion paper, Racing to Ruin. The same summary describes Koh as also a research scientist at Google DeepMind, an affiliation not listed on the paper. I found no policy documents or critiques citing it yet; later uptake is plausible, but that is a forecast.
Discussant
Why would funders, regulators or labs care, and why might an Unjournal evaluation be worthwhile?
Host
Labs and the people behind pacing statements are implicitly betting that voluntary restraint can be self-enforcing; this paper says that depends on the hazard from running ahead of safety relative to the profit growth from doing so, and that it is fragile to hidden progress. Policymakers get a formal argument that observability of internal capabilities widens the conditions for restraint. Funders weighing safety research against governance work get the result that faster safety progress helps but does not by itself remove the racing equilibrium. An evaluation is useful because the results will likely be cited loosely, the proofs are long, and most of the judgment lies in mapping parameters to the real world. It pairs naturally with the static AGI race paper.
Discussant
Then let me lay out the critiques. First, safety is exogenous. The safety threshold rises at a fixed rate regardless of what firms do. In practice labs fund much of the alignment work, and the paper's own comparative static says faster safety helps. A model where firms split effort between capabilities and safety could change the regions substantially.
Host
Agreed, and the paper does not model safety investment.
Discussant
Second, the threshold and the hazard are common knowledge. In reality, disagreement and uncertainty about where danger begins is the hard part; with heterogeneous beliefs, a grim trigger could fire by mistake, or firms could exploit ambiguity. Third, two symmetric firms, no entry, no open-weight releases. A footnote suggests the results might apply to competition between the United States and China, but there are several frontier labs, and pacing with more players is usually harder. Fourth, there is no development cost and capability is a scalar, so compute constraints, a main real-world brake, are absent.
Host
And the functional form?
Discussant
Fifth. Exponential profits make the problem stationary, which yields the clean cutoffs; Proposition three relaxes this, but the three-region theorem relies on it. Sixth, equilibrium selection. The interesting region has both equilibria, and the subgame perfect analysis admits riskier outcomes that firms prefer. The paper shows when pacing can be sustained, and gives no reason to expect it will be. Seventh, the boundary conventions. Since pacing means sitting exactly on the safety line, an evaluator should ask whether the results survive the Filippov alternative or valuing a profile by its worst admissible path.
Host
What else should an evaluator check?
Discussant
Whether any plausible calibration puts us in one region or another; none of the relevant numbers is known, so a sensitivity exercise showing what annual hazard rates would be needed would help. Whether the Theorem two bound is tight enough to mean anything. Whether the transparency conclusion survives if hidden internal models generate hazard. The competition angle: self-enforcing pacing is also coordinated output restriction, and the paper mentions critics worried about incumbent rents. And a careful check of the appendix proofs, especially the Markov best-response lemma and the recursive guarantee. This is version one, with small typos.
Host
In short: a rigorous continuous-time model giving clean conditions for when profit-maximizing labs can sustain restraint, showing that hidden internal progress makes restraint harder and that faster safety research alone does not rule out racing. Its policy relevance rests on known, exogenous safety and two symmetric firms, assumptions an evaluator should examine closely.

Sources