Risk Ceilings and Deadlines

Interactive companion to the paper

Risk Ceilings and Development DeadlinesPacing AI under Uncertain Safety Productivity

Proposals to pace frontier AI promise a ceiling on the monthly hazard of catastrophe and a deadline on development delay. Both rest on a number nobody knows: how productive safety research is. This paper asks when the two promises can be kept together, when safety research should happen, and what customers give up.

No productivity bound, no guaranteed ceilingAt the fitted productivity, the scenario’s ceiling takes at least 17.2 months of all-out safety research, and the 12-month deadline allows 30. With fixed hazards, no rule keeps both promises if productivity can fall below 0.57 of the fitted value, the necessary bound.Section 5.1, Proposition 1
Learn first, then replay development lateAfter a year of all-out safety research, growing compute lets the lab replay the full-speed path with 12.5% of compute for research. At the scenario’s ceiling, with fixed hazards, this spare-compute replay needs a productivity bound only 21% above the necessary bound, and there it leaves 8% catastrophe risk against 75% at full speed.Sections 5.2–5.5 and 7, Propositions 2 and 3
Measuring safety research lowers the priceThe spare-compute replay costs customers 13% of five-year services. A regulator who can rule out productivity below the fitted value can choose a replay that costs 4.1%.Sections 7 and 9, Figure 5

Each step below follows a section of the paper. The figures run the paper’s model in your browser and open at the paper’s settings, where every number equals the one the paper prints. Move a slider to see what drives a result; “Paper’s settings” puts it back. The longer searches of Steps 7, 9, 10, 11 and 13 run when you reach them. The scenario’s numbers show what the AI Futures Project’s judgments imply; they are not estimates. The paper takes its 75 percent full-speed catastrophe probability from that example and runs lower values down to 5 percent; the bounds do not change (Step 2).

Contents
  1. Five findings at a glance
  2. The pieces
  3. Compute and the takeoff period
  4. Hazard and knowledge
  5. The fitted productivity
  6. Keeping both promises
  7. The two promises
  8. What a ceiling requires
  9. Learn, then replay
  10. How close the bounds are
  11. When to learn
  12. The value of a month
  13. When to learn first
  14. Timing that keeps both promises
  15. Who pays, and fixed rules
  16. What customers give up
  17. When research does nothing
  18. Fixed rules across productivity
  19. Conclusion
  20. What a regulator can promise

At a glance · Paper, Introduction

Five findings, and the one calculation behind them

A pacing proposal makes two promises: the monthly hazard of catastrophe stays below a ceiling, and no capability milestone arrives more than a year late. Whether both can be kept depends on how productive safety research is, and nobody knows that. The diagram below shows the calculation that ties the two promises to this one number. The fourteen steps that follow build it piece by piece; here is where they lead.

A way to picture it: a race car. The engine is capability research and the brakes are safety research. A ceiling on hazard limits how risky each stretch of track may be, and the deadline is the lap time. Keeping to both needs good enough brakes. If the brakes might be weak, no plan keeps the limit and the lap time together. The best plan has the mechanics build the brakes first and then has the car run the lap with a stronger engine, since compute keeps growing.

  1. Keeping both promises requires a minimum productivity of safety research, the necessary bound. The final model’s unprotected hazard is about 57 times the ceiling, so safety knowledge must halve it about 5.8 times before the takeoff period ends. At the fitted productivity that takes at least 17.2 months of all-out safety research, while development may last at most 30 months, 18 at full speed plus the 12-month lead. So no rule keeps both promises if productivity can be below 0.57 of the fitted value. Steps 4–5
  2. Learning first and then replaying development comes close to that minimum. The lab first puts all compute into safety, then follows the full-speed path a year late. Because compute keeps growing, the replay needs only 12.5 percent of compute for research, and the spare-compute replay puts the rest into safety. Its replay bound is 21 percent above the necessary bound, and at that bound it holds catastrophe risk to 8 percent, against 75 percent at full speed. The gap widens when early knowledge loses value as models grow stronger or when later research is more productive. Steps 6–7
  3. Earlier months of learning are worth more, unless stronger models make safety research more productive and early knowledge fades at the same time. A month of delay is worth the hazard its learning removes minus the hazard of spending longer at each capability, and late in the takeoff period the second outweighs the first. With the scenario’s technology, a 12-month pause with safety research at the start cuts the probability of catastrophe by 60 points, and the earlier the pause, the better. Steps 8–10
  4. Customers pay for the guarantee. Services depend on both inference compute and model quality, so compute moved to safety research costs customers either way: it lowers their services now if it comes from inference, and it delays better models if it comes from capability research. The spare-compute replay gives up 13 percent of five-year service value. Evidence about productivity lowers the price: if the regulator can rule out productivity below the fitted value, the cheapest policy in the replay class gives up only 4.1 percent. Step 11
  5. Rules that fix compute shares make a different promise. They fix development dates and let risk vary with safety productivity, while a ceiling fixes hazard and lets delay adjust. Across ten versions of the safety technology, the better options are rules that move compute into safety early: compute floors lower the probability of catastrophe by 15 to 71 points, and a pause with safety research by 21 to 68. The ceiling and the R&D-model lag delay development late in the takeoff period, and their effect is not consistent: they can raise risk as well as lower it. Steps 12–13

How to read the diagram. The ceiling sets how much safety knowledge the lab must have before its most dangerous models. The deadline sets how long it has to build that knowledge, and productivity sets how fast it can. Dividing the knowledge needed by the fastest learning gives a minimum time; when that time is longer than the deadline allows, no rule keeps both promises. Steps 4 and 5 compute each piece, and Steps 6 and 7 show how close a practical policy gets.

The implication is that a regulator’s first need is evidence on how productive safety research is. Without a productivity bound, no rule can promise both; with a bound at or above the replay bound, learn-then-replay keeps both, at a service cost the regulator can state in advance.

Terms in plain words

Hazard
the chance per month, for a short stretch, that a catastrophe, an irreversible loss of human control over AI, happens if it has not happened yet
Ceiling
the cap on monthly hazard the regulator promises; about 1.6 percent a month per company in the scenario
Deadline and lead
the deadline caps the delay of every capability milestone at the lead; 12 months in the scenario, the time a rival would need to catch up
State and step hazard
the hazard of running the current model, which grows with the months spent at each capability, and of training a stronger one, which depends only on how far the frontier moves
Safety knowledge
what safety research builds; it scales down the hazard of every later model, and each ln 2 of it halves that hazard
Safety productivity
how much knowledge a unit of safety compute buys. Nobody knows it; the fitted value makes a 12-month pause at the start, standing for a safety-case regime, bring catastrophe risk to 40 percent
Productivity bound
the lowest productivity the regulator cannot rule out; a guarantee must hold at every productivity above it
Necessary and replay bounds
the smallest productivity bound that passes the test every rule must pass, and the smallest at which learn-then-replay keeps both promises; the true threshold lies between them
Learn-then-replay
all compute to safety first, then the full-speed path, followed late; the spare-compute replay also puts all compute the replay’s research does not need into safety
Fixed and normalized hazards
two ways to vary productivity: fixed hazards keep the hazard functions, as in the propositions; normalized hazards rescale them so that full speed keeps 18 months and 75 percent, as in the rule comparisons

Part 1The pieces: compute, hazard and the fitted productivity

Step 1 · Paper, Sections 2.1, 3.1, 4.1 and 4.2

Compute grows fast, so a late start can replay full speed with a small research share

The model follows one frontier lab through a takeoff period. It starts at the Automated Coder, an AI that fully automates software engineering, and ends at top-expert-dominating AI, which dominates top human experts at virtually all cognitive tasks. Capability is an index from 0 at the first milestone to 1 at the second. The lab divides its compute among customers, capability research and safety research. The milestones, and the 18 months a full-speed takeoff period takes, come from the worked example of the AI Futures Project.

Compute doubles every 6 months, the trend in training compute since the early 2010s (Sevilla et al. 2022), so it grows 4-fold in a year and 8-fold over the 18-month full-speed takeoff period. Better models do research faster, so research speed rises 28-fold over the takeoff period. Capability reaches one half at month 15.1 and covers the upper half in the last 3 months.

Growing compute gives the paper its main policy. A lab that starts τ months late can follow the full-speed path exactly with the research share x0e−Gτ:

x0e−Gτ K(τ + u) = x0K(u) for all u ≥ 0

Both sides are research compute at the same stage u of the path, so the late lab reaches every milestone exactly τ months late. Compute quadruples in a year, so a lab that starts a year late needs one eighth of its compute for research instead of one half. The rest is free for other uses.

Where the baseline numbers come from

NumberValueSource
Full-speed takeoff period, Automated Coder to top-expert-dominating AI18 monthsAI Futures Project (2026), worked example: 1.5 years.
Probability of catastrophe at full speed75% (range 5 to 75%)AI Futures Project (2026), worked example. The ceiling is re-solved at each probability, so the bounds do not change (Step 2).
Probability of catastrophe under a safety-case regime40%AI Futures Project (2026). Treating it as a 12-month pause at the start, which fixes the safety yield, is the paper’s assumption.
Lead over the closest rival; the rival’s catch-up time4–8 months; about a yearAI Futures Project (2026); the paper’s 12-month deadline follows from it.
Example risk cap; frontier companies1% a month per company; 3AI Futures Project (2026). The scenario’s ceiling, about 1.6% a month per company, uses exactly the 12-month lead at the fitted productivity.
Compute floors; research-model lag (examples)70% customers, 25% safety; 9 monthsAI Futures Project (2026).
Growth of computedoubles every 6 monthsSevilla et al. (2022): training compute has doubled about every six months since the early 2010s.
Full-speed compute shares50% research, 5% safety, 45% customersAn illustrative assumption of the paper.
Research speed at the endpoint, same compute10 times the startAn illustrative assumption of the paper.
Service quality at the endpoint; discount rate10 times the start; 3% a yearIllustrative assumptions of the paper.

Capability over time

Compute and research compute

Left panel, capability. The gray dashed curve is the full-speed path, from 0 at the Automated Coder to 1 at top-expert-dominating AI in 18 months. The blue curve starts the same path the number of months on the slider later. Both creep up for a year and then accelerate, because compute keeps growing and each gain speeds up the research that produces the next.

Right panel, compute. Compute in multiples of the initial stock, on a log scale. The green line is total compute. The gray dashed line is full-speed research compute, half of the total. The blue line is the late lab’s research compute: the same amounts, the same number of months later. The gap between the blue and green lines is the compute the late lab has free.

The implication is that waiting is cheap in compute: the late lab replays full speed with a small research share and can put the rest into safety research. Steps 6 and 7 build the main policy on this.

The math

Step 2 · Paper, Sections 2.2, 3.2 and 4.2, and the Supplemental Appendix

Hazard comes from the model’s state and from each step, and safety knowledge scales both down

The catastrophe is an irreversible loss of human control over AI. The paper reads this danger as misalignment. A more capable model is more likely to pursue goals its developers did not intend and better able to escape control, so the danger rises with capability. The paper treats the catastrophe as a hazard: the chance per month that it happens now, given that it has not happened yet. A cumulative hazard H gives a probability 1 − e−H, so the scenario’s 75 percent at full speed is a cumulative hazard of ln 4 ≈ 1.386. The paper counts hazard only during the takeoff period, as the AI Futures Project’s 75 percent does. So it assumes that the danger ends at completion, for example because a top-expert-dominating AI still under control can help align its successors.

State hazard is the danger of having a model of a given capability in the world; it accrues per month. Step hazard is the danger of training a stronger model; it accrues per unit of capability gained. Safety research builds a stock of knowledge S that scales both down at every later level:

λ(t) = [h(F) + k(F) Ḟ] e−S,   Ṡ = a σ̄ y(s)

Adding ln 2 to S halves the hazard. The productivity a is the paper’s central unknown. At the fitted productivity, a = 1, a month of research at the usual share removes 8.3 percent of every later hazard. Here productivity varies with fixed hazards: h and k stay at their scenario values, so a lower productivity means only that research produces less knowledge.

Even with the knowledge it builds, the full-speed hazard rises from 0.0047 a month at the start to 1.20 at the end. It exceeds the scenario’s ceiling of 0.0492 a month from month 13.1 on. A constant hazard at the final rate would mean about 70 percent risk within one month; at the ceiling, 4.8 percent. Without any knowledge, the final hazard would be 5.6 a month, half of it state hazard, h(1) = 2.82.

The ceiling of 0.0492 a month belongs to the 75 percent judgment. The paper sets the ceiling so that it limits delay to the 12-month lead, and it normalizes the hazard to the full-speed probability. At a lower probability the hazard scales down in proportion and the matching ceiling scales with it, so the ratio of the final hazard to the ceiling stays the same, and so do the bounds. The Supplemental Appendix reruns the scenario at full-speed probabilities of 25, 10 and 5 percent, with the pause judgment cutting hazard in the same proportion as at 75 percent, and gets the same necessary bound, 0.57, and the same replay bound, 0.69, with matching ceilings of 0.0102, 0.0037 and 0.0018 a month. Nothing in this paper therefore breaks at the lower end of the range. A ceiling held at 0.0492 while the probability falls would be a much looser promise, and the paper does not study it.

Hazard per month at full speed

Catastrophe probability over the takeoff period

Left panel, hazard. Hazard per month along the full-speed path, on a log scale. The blue curve includes the knowledge the lab builds at the productivity on the slider; the gray dashed curve has none. The dotted line is the scenario’s ceiling of 0.0492 a month. The hazard rises steeply with capability: three quarters of the full-speed cumulative hazard comes after capability passes one half.

Right panel, probability. The cumulative probability of catastrophe on the same paths. At the fitted productivity it ends at 75 percent. A lower productivity builds less knowledge, so the blue curves move toward the gray ones.

The implication is that full speed breaks the ceiling late in the takeoff period, when the models are strongest. Keeping the hazard below the ceiling takes knowledge, and knowledge takes time.

The math

Step 3 · Paper, Sections 4.1 and 4.2

The yield is fitted to a 40 percent outcome, and learning has diminishing returns to compute

The scenario follows the AI Futures Project’s worked example. At full speed the takeoff period takes 18 months and carries a 75 percent probability of catastrophe. A safety-case regime, in which third-party assessors estimate each company’s risk and the government caps it, cuts that to about 40 percent while staying ahead of the rival. The paper maps that regime to a 12-month pause at the start, with the usual safety share and compute standing still. This mapping is a modeling assumption.

The fitted yield σb solves

h(0)(1 − e−12σb)/σb + e−12σb [−ln(1 − 0.75)] = −ln(1 − 0.40)

The first term is exposure during the pause, which starts at h(0) = 0.00235 a month and falls as knowledge grows. The second is the full-speed cumulative hazard, 1.386, scaled down by the pause’s knowledge. The solution is σb = 0.086116 a month. The pause multiplies all later hazard by 0.356, cutting 1.386 to 0.493. Waiting adds only 0.018, because the initial model has a low hazard. The total, 0.511, is a 40 percent probability.

Learning also depends on the share of compute in safety research, through the multiplier y(s) = ψ + (1 − ψ)(s/s0)α with ψ = α = 1/2. Half of learning needs no dedicated compute. Moving all compute into safety multiplies the share by 20 but learning only by 2.74. A pause that moves the research share into safety learns 2.16 times as fast; an all-inference pause learns at half speed.

Cumulative hazard of a pause at the start

The learning multiplier

Left panel, the fit. Cumulative hazard of a pause at the start with compute standing still, against the pause’s length, at the fitted yield. The orange curve is exposure while paused; the blue curve is the full-speed hazard after the pause, scaled down by its knowledge; the green curve is their sum. The dotted line is the 40 percent target. The fit makes the green curve cross it at 12 months.

Right panel, the multiplier. Learning relative to the usual safety share, against the share of compute in safety research. It rises fast at first and then slowly: dedicated compute has diminishing returns.

The implication is that the scenario’s productivity, a = 1, is a fit to judgments, not an estimate. The rest of the page measures productivity relative to it.

The math

Part 2Keeping both promises: what a ceiling requires and how close a policy gets

Step 4 · Paper, Sections 2.3, 3.3, 4.3 and 8.3

Two promises: a ceiling on hazard and a deadline on delay

The AI Futures Project lists four ways to pace a frontier lab: a temporary pause, compute floors, a lag on the models used for research, and a maximum risk threshold enforced by third-party assessors. Amodei’s case for using the extra time for safety work suggests a pause in which freed research compute goes to safety. The paper studies six rules, each written as a constraint on how compute is used.

Six rules for pacing a frontier lab (Table 1)

RuleConstraint while it applies
Compute floorsMinimum shares for customers and for safety research; capability research gets the rest.
Pause, all inferenceNo compute for capability or safety research; all of it serves customers.
Pause with safetyNo dedicated capability research; the freed compute goes to safety research.
R&D-model lagCapability research uses the model of some months earlier.
Safety floorA safety share drawn from customer compute; capability research unchanged.
CeilingThe hazard may not exceed the ceiling; when it binds, the lab slows just enough and moves the freed compute into safety research.

A ceiling caps the monthly hazard. The scenario’s ceiling is 0.0492 a month, about 1.6 percent a month per company for three companies, close to the Project’s example of 1 percent. A deadline caps the delay at every milestone, because a rival could overtake at any capability level. The lead used by a path is its largest milestone delay, and the deadline requires it to be at most L = 12 months. The ceiling is set so that it uses exactly that year at the fitted productivity.

A promise is a guarantee if it holds at every productivity the regulator cannot rule out; the lowest such productivity is the productivity bound. A ceiling fixes hazard and lets delay absorb the uncertainty. With normalized hazards, which keep full speed at 18 months and 75 percent at every productivity, the ceiling uses 25 months of lead at half the fitted productivity and 52 at a quarter.

Hazard per month under the ceiling

Milestone delay under the ceiling

Left panel, hazard. Hazard per month at full speed (gray dashed) and under the ceiling (blue), on a log scale, at the productivity on the slider. Once the full-speed hazard reaches the ceiling, the lab slows down so the hazard stays exactly at the ceiling, while the freed compute builds knowledge.

Right panel, delay. How many months late each capability level arrives under the ceiling. The dotted line is the 12-month deadline. The largest delay is the lead used.

The implication is that the ceiling keeps its hazard promise at every productivity, but its delay grows as productivity falls. A regulator who promises both must be able to rule out low productivity.

The math

Step 5 · Paper, Section 5.1, Proposition 1

A ceiling requires a minimum stock of knowledge, and the deadline limits the time to build it

Four steps bound the time any path needs to respect a ceiling c̄ below the final model’s unprotected state hazard h(1).

  1. The final model must be under the ceiling. Step hazard is never negative, so the ceiling requires h(1)e−S(T) ≤ c̄, or S(T) ≥ ln[h(1)/c̄].
  2. Knowledge grows at a bounded rate: at most aσ̄maxȳ a month, with all compute in safety at the most productive capability.
  3. Combining the two, the takeoff period must last at least T ≥ ln[h(1)/c̄]/(aσ̄maxȳ) months.
  4. The final milestone may arrive at most L months late, so T ≤ T0 + L. If the minimum duration is longer, no path keeps both promises.

This is Proposition 1. In the scenario, h(1) = 2.82 a month is about 57 times the ceiling, so the lab needs S(T) ≥ 4.05, about 5.8 halvings. The fastest learning is 0.236 a month, so one halving takes 2.9 months and compliance takes at least 17.2 months. That fits within 18 + 12 = 30 months, but not once productivity falls below 17.2/30 ≈ 0.57 of the fitted value.

That number is the necessary bound. A regulator who cannot rule out productivity below it cannot promise this ceiling and this deadline together, whatever rule it chooses. The condition is generous: it lets all compute go to safety for the whole time, leaving none to develop capability. So it is necessary but not sufficient.

Minimum compliance time against productivity

The necessary bound against the ceiling

Left panel, compliance time. The shortest takeoff period that respects the ceiling on the slider, against productivity on a log scale. The dotted line is the 30 months the deadline allows. In the shaded range the curve lies above it, so no rule keeps both promises.

Right panel, the necessary bound. The lowest productivity at which the ceiling can fit within 30 months, against the ceiling on a log scale. A tighter ceiling needs more knowledge, so the bound rises as the ceiling falls. At h(1) = 2.82 no knowledge is needed. The dotted line marks the scenario’s ceiling.

The implication is that a ceiling can be guaranteed only together with a productivity bound. If productivity can be arbitrarily small, every ceiling below h(1) fails in some state of the world.

The math

Step 6 · Paper, Sections 5.2 and 5.3, Proposition 2

Learn, then replay: a policy that keeps both promises when productivity is not too low

The policy has two phases. First the lab learns: for τ months all compute goes to safety research, and capability stays at the Automated Coder. Then it replays the full-speed path τ months late. By the replay identity it needs only the research share x0e−Gτ. The regulator chooses how much of the rest goes to safety, the replay safety share sr, and customers get the remainder. Every milestone arrives exactly τ months late, so the policy uses τ months of lead.

Three steps show when it keeps both promises at every productivity a ≥ amin. While the lab learns, the hazard is the initial model’s state hazard, falling as knowledge grows, so the learning phase passes if c̄ ≥ h(0). On the replay, hazard falls as productivity rises, so the worst case is a = amin. And a longer learning phase only helps, so a full year works if any learning time does. Proposition 2 states the exact condition:

c̄ ≥ h(0) and B(sr) e−aminσ̄(0)ȳL ≤ c̄

Here B(sr) is the largest replay hazard at amin before counting the knowledge from the learning phase, and the exponential is the protection that L months of learning provide. The smallest amin that passes is the replay bound. With a year of learning and all spare compute in safety, the spare-compute replay, it is 0.69. With the usual safety share it is 1.08, above the fitted productivity itself.

At its bound, the year of learning builds 1.96 of knowledge and the replay adds 2.78. The total, 4.74, is exactly what the last milestone needs, so the hazard touches the ceiling there. Full speed at the same productivity breaks the ceiling from month 12.0 on.

Capability: learn, then replay

Hazard per month

Left panel, capability. The policy’s capability (blue) stays at zero while the lab learns and then follows the full-speed path (gray dashed) exactly the learning time later.

Right panel, hazard. Hazard per month on a log scale. During learning it is the initial model’s state hazard, falling as knowledge accumulates. When development resumes, step hazard starts again, so the hazard doubles, and then it rises with capability. At the policy’s bound it touches the ceiling (dotted) exactly at the last milestone. Full speed at the same productivity (gray dashed) breaks it from month 12.0.

The implication is that spare compute is the regulator’s lever. Putting it in safety during the replay lowers the bound from 1.08 to 0.69, at a cost to customers that Step 11 measures.

The math

Step 7 · Paper, Sections 5.4 and 5.5, Proposition 3

The two bounds nearly coincide when compute grows fast and early knowledge keeps its value

The necessary bound imagines that learning uses all compute for the whole time; the replay must also spend compute on research, so it learns more slowly. When all hazard is state hazard and the yield is constant, Proposition 3 brackets the threshold a*, the lowest productivity bound at which some rule keeps both promises:

ln[h(1)/c̄]/[σ̄ ȳ(T0 + L)] ≤ a* ≤ ā = ln[h(1)/c̄]/[σ̄(ȳL + yrT0)]

The two bounds differ only in the learning each allows. In the state-hazard version of the scenario, a year of compute growth cuts the replay research share to 12.5 percent, so the replay learns at yr = 2.59 against ȳ = 2.74. Their ratio is 1.033: the bounds differ by 3.3 percent. Compute growth drives this. With a 12-month doubling time the gap is 7.0 percent, with 24 months 10.6 percent, and with no growth 17 percent.

Step hazard widens the bracket. In the scenario, state and step hazard are equal along the full-speed path, so the replay must hold one more halving of knowledge than the necessary bound counts. At the scenario’s ceiling that adds 17 percent, and the replay bound is 21 percent above the necessary bound. At looser ceilings, up to about 1 a month, the gap grows to 72 percent. Rising yields or lost knowledge widen it further, because the policy relies on knowledge learned on the initial model.

Only state hazard

The scenario: half state, half step hazard

The replay bound relative to the necessary bound, at the scenario’s ceiling

Left panel, state hazard only. Productivity bounds against the ceiling, on a log scale, in a version of the scenario refitted with only state hazard. The black line is the necessary bound, below which no policy keeps both promises. The blue line is the spare-compute replay’s bound and the orange line the usual-share replay’s. The blue and black lines nearly coincide. The dotted line marks the scenario’s ceiling, and the dots the ceiling on the slider.

Right panel, the scenario. The same bounds with the scenario’s even split between state and step hazard. The replay must also absorb the step hazard of the last milestones, so the blue line sits above the black one, more so at loose ceilings.

The table. The ratio of the spare-compute replay bound to the necessary bound at the scenario’s ceiling, when the yield triples between the first and last milestones or half of knowledge is lost as capability rises. Gray rows print the paper’s values. Because the necessary bound is loose in both cases, these ratios bound the policy’s shortfall rather than measure it.

The implication is that learning first is cheap when compute grows fast and early knowledge protects later models. Where the two bounds differ, the paper’s results neither rule out a policy nor provide one.

The math

Part 3When to learn: the value of a month and the timing of pauses

Step 8 · Paper, Sections 6.1 and 6.2, Proposition 4

A month of delay is worth its learning minus its exposure, and earlier months are worth more

Feasibility does not settle when to learn. The paper first isolates time with a benchmark: after a pause the lab resumes exactly the path it would have followed, milestone by milestone. Two quantities then measure the trade-off: the hazard still ahead, A(t), and the hazard of standing still, w(t) = h(F)e−S. A month of pause adds m of knowledge, which saves about mA(t) of the hazard ahead, and costs w(t) of exposure:

v(t) = m A(t) − w(t)

Proposition 4 gives the exact effect of a pause of δ months: ΔH = −[(1 − e−mδ)/m] v(t). The factor in brackets counts the pause in effective months; it is less than δ because exposure shrinks as knowledge accumulates. The fit of Step 3 is this at work. At the start, v = 0.086116 × 1.386 − 0.00235 ≈ 0.117, and a 12-month pause counts as 7.48 effective months, so it saves about 0.875.

Along the full-speed path the value of a month falls with the start date. At the usual share it turns negative at month 15.9. A pause that moves research compute into safety learns 2.16 times as fast: its value starts at 0.255 and stays positive until month 16.7. An all-inference pause stops paying at month 14.8. Late in the takeoff period the current model is already dangerous, and little hazard is left for new knowledge to protect.

The value of a month by start date

Benefit and cost of a month at the usual share

Left panel, value of a month. Cumulative hazard saved per month of pause, against the date the pause starts, for three safety shares during the pause: the usual share (blue), research compute moved into safety (green) and all compute to customers (orange). Above zero a pause lowers risk; below, it raises it. The dots mark the start date on the slider.

Right panel, the two terms. The two parts of the value at the usual share, on a log scale. The benefit of learning, mA(t), falls as less hazard remains ahead. The cost of exposure, w(t), rises with capability. They cross at month 15.9.

The implication is that in the benchmark the best month to learn is the first. A yield that rises with capability can reverse this; Step 9 compares finite pauses exactly.

The math

Step 9 · Paper, Section 6.3, Proposition 5

Learning later pays only when research gets more productive and early knowledge fades

Learning first need not be best. Stronger models can make safety research more productive, and knowledge learned later has less development over which to lose value. Waiting for those gains leaves earlier hazards unprotected. For this comparison, knowledge may lose value as capability advances: knowledge carried across a capability gain ΔF keeps the fraction e−dΔF. A pause of δ months at date t then saves

𝒱δ(t) = ∫ λ(u)[1 − e−m(t)δR(t,u)] du − w(t) gδ(m(t))

The first term is future hazard removed, with R(t, u) the share of the pause’s knowledge still useful at date u; the second is hazard incurred while learning. Proposition 5 compares two start dates. Waiting pays only if the later pause’s extra protection outweighs both the protection forgone before it and its extra exposure.

Table 3 runs the comparison for a 12-month pause with safety, after which the lab replays the remaining path. Hazards are normalized so each row’s reference keeps 75 percent. With the scenario’s constant yield and full transfer, learning first is best: a pause at the Automated Coder cuts the probability of catastrophe by 60 points, to 15 percent. A yield that triples does not change that, and half transfer alone moves the best start only to month 2.1. Together they move it to month 9.2, and the pause cuts the probability by 47.9 points instead of 44.8.

Fall in catastrophe probability by start date

When developing further makes a safety pause more valuable (Table 3)

The chart. The fall in the probability of catastrophe, in percentage points, from a 12-month pause with safety, against the date the pause starts. The gray dashed curve is the scenario’s technology; the blue curve uses the yield rise and retention on the sliders. The dot marks the best start.

The table. The paper’s Table 3, recomputed here, with a last row for the technology on the sliders. Gray rows print the paper’s values.

The implication is that waiting is worth considering only when both forces are present, and even then the gain is small: 47.9 points against 44.8 in the paper’s last row.

The math

Step 10 · Paper, Section 6.4, Proposition 6

A useful pause must also keep both promises

Lower cumulative hazard does not by itself keep the monthly hazard below the ceiling. Three checks identify every pause in this class that keeps both promises (Proposition 6).

  1. Check the path before learning. A pause leaves its history unchanged, so if the reference hazard has already exceeded the ceiling, starting then is too late.
  2. Check the learning period. The hazard during the pause starts at w(t) and falls, so the pause meets the ceiling exactly when w(t) ≤ c̄.
  3. Protect the replay within the lead. The pause must last at least δc(t), the learning that brings every resumed hazard below the ceiling, and at most L.
δc(t) = supu [ln(λ(u)/c̄)]+ / [m(t) R(t, u)]

Take the technology of Table 3’s last row, with a tripling yield and half transfer. At the scenario’s ceiling of 0.0492 a month, no start works: even an immediate pause would need 35.5 months of learning (17.2 with the scenario’s own technology, a different quantity from the 17.2-month minimum of Step 5, which assumes learning at the fastest possible rate throughout), and the lead allows 12. At a ceiling of 0.15 a month, only starts between months 13.5 and 15.8 pass. Earlier starts fail the duration test: the yield is still low and the knowledge would fade. Later starts fail the first test. The best feasible start is month 13.5, later than the unconstrained best, and it cuts the probability by 43.2 points. At 0.30 a month, starts from month 8.5 pass; at 0.50, every start before month 17.2.

The starts that pass

The duration test

Left panel, the starts that pass. The fall in catastrophe probability from a 12-month pause with safety, by start date, for the technology with a tripling yield and half transfer. The green band marks the starts that pass all three tests at the ceiling on the slider; the orange band marks starts after the reference has already broken the ceiling.

Right panel, the duration test. The minimum learning time by start date. The dotted line is the 12-month lead. Starts where the curve is above the line fail the third test.

The implication is that the ceiling can move the best pause later as well as earlier. First find the dates that keep the promise, then compare the protection they buy.

The math

Part 4What customers pay, and rules that fix compute

Step 11 · Paper, Section 7

Customers pay through less compute and later improvements, and evidence lowers the price

Customers receive services worth √(eμFiK) a month: inference compute iK, with quality multiplied by eμF. Quality rises tenfold over the takeoff period. The service loss of a policy is the percentage fall in discounted service value over q months, relative to full speed:

Jq = ∫0q e−rt √(eμF iK) dt

The replay safety share trades service value against the productivity bound. The spare-compute replay needs a bound of 0.69. It gives up 13 percent of five-year service value and all of two-year value, because customers get nothing until the takeoff period ends at month 30. At its bound it holds the probability of catastrophe to 8 percent, against 75 percent at full speed. The usual-share replay gives up only 4.7 and 51 percent, but needs a bound of 1.08.

Stronger evidence lowers the price. For each productivity bound, the figure chooses the learning time and the replay safety share that minimize the five-year loss while respecting the ceiling at the worst productivity. At the fitted productivity, the cheapest policy learns for 7.5 months, then gives 21.0 percent of compute to research, 40.1 percent to safety and 38.9 percent to customers. It gives up 4.1 percent of five-year value, against 5.0 percent with a year of learning. It also leaves more risk at the fitted productivity: 8 percent, against 2 percent for the spare-compute replay.

Service value and risk are reported side by side; the paper does not convert one into the other. Budgets need a stated horizon: full-speed service value over five years is 13.65 times that over two.

The price of a guaranteed ceiling and deadline

Compute during the replay

Left panel, the price. Five-year service loss of the cheapest replay policy that guarantees the scenario’s ceiling and the 12-month deadline, against the productivity bound. Blue: learning time and replay safety share both chosen. Gray dashed: learning fixed at 12 months, replay safety share chosen. Left of the black line, the necessary bound, no policy keeps both promises; between it and the start of the curves, no replay policy does.

Right panel, compute. How each policy divides compute once development resumes, with its five-year service loss at the end of each bar. The spare-compute replay leaves customers nothing until the takeoff period ends.

The implication is that the binding input is the productivity bound. Evidence that rules out low productivity makes the joint promise cheaper.

The math

Step 12 · Paper, Sections 8.1 and 8.2, Proposition 7

When research does nothing, slowing down raises risk, but an early pause can lower it

Compute floors, pauses and lags fix compute allocation instead of hazard. Such a rule produces the same capability path at every productivity, because the law of capability contains neither knowledge nor productivity. So these rules fix the dates and let risk absorb the uncertainty, while a ceiling fixes hazard and lets delay absorb it: prices versus quantities.

The sharpest case is research that produces no knowledge at all. A constant research share below the full-speed one makes every milestone arrive M = (x0/x)ε times later on the research-capacity scale, so the lab spends more time at every capability level. Step hazard is unchanged and state hazard is larger, so cumulative hazard rises (Proposition 7). The scenario’s compute floors fix research at 7.7 percent, so M ≈ 2.5 and the takeoff period ends at month 30.0. When research does nothing, they raise the probability of catastrophe by 5.4 points, and the R&D-model lag raises it by 16.8.

A pause is different, because compute keeps growing while the lab waits. It resumes with more compute and crosses each later level faster than at full speed. The exact change in takeoff hazard is

ΔH = exposure while waiting − exposure saved after resuming

A 12-month pause at the start adds 0.0282 of exposure while waiting, because the initial model has low state hazard, and saves 0.201 afterward. So ΔH = −0.173, and the probability falls by 4.72 points. The two terms are equal for a pause starting at month 12.8; a later pause raises risk. This assumes step hazard is proportional to speed. If it rose faster, catch-up after a pause could add risk: the overhang worry.

The two terms of a 12-month pause

Change in catastrophe probability when research does nothing

Left panel, the two terms. Exposure added while waiting (orange) and exposure saved after resuming (blue) for a 12-month pause, against its start date, on a log scale, when research produces no knowledge. The saved term changes little with the start date; the waiting term grows with the hazard of the held model. They cross at month 12.8.

Right panel, four rules. Change in the probability of catastrophe, in percentage points, when research does nothing. Rules that slow research for the whole takeoff period add exposure. A pause early in the takeoff period lowers risk because the lab catches up faster later; one after month 12.8 raises it. A safety floor changes nothing, because it leaves capability research alone.

The implication is that slowing down is not safe in itself. What it buys depends on safety productivity, and with none, only an early pause helps.

The math

Step 13 · Paper, Section 8.3

Allocation rules fix dates, and early safety research lowers risk most consistently

The paper holds each rule at its one-year-lead setting and varies productivity from a tenth to twice the fitted value, with normalized hazards. Compute floors fix research at 7.7 percent and safety at 25 percent. The two 12-month pauses start at months 1.5 (with safety) and 10 (all inference). The R&D-model lag is 21 months, and the safety floor uses no lead. These are fixed settings for comparison, not each rule’s best design.

The allocation rules keep their dates as productivity changes. The ceiling uses 25 months of lead at half the fitted productivity and 52 at a quarter. Every month at the ceiling carries hazard, so a long delay can leave more risk than full speed, as the bars show at low productivity.

The paper also retunes each rule in ten versions of the safety technology. Rules that move compute into safety research early lower risk most consistently: floors lower the probability of catastrophe by 15 to 71 points across the versions, and the pause with safety by 21 to 68. Rules whose delay comes late in the takeoff period are not consistent. The ceiling’s effect ranges from a 24-point rise to a 74-point fall, and the lag’s from a 12-point rise to a 36-point fall. Late delay pays when safety research grows more productive with capability or with compute, or when danger continues after the takeoff period.

Lead used against productivity

Change in catastrophe probability from full speed

Left panel, lead used. The largest milestone delay of the ceiling (orange) and of compute floors (blue) against productivity, on a log scale, with normalized hazards. Floors, both pauses and the lag use the deadline’s 12 months at every productivity; the ceiling uses more as productivity falls and less as it rises. The gray dashed line is the least delay any rule needs to keep the ceiling (Step 5); above about the fitted productivity, none is needed.

Right panel, the six rules. The change in the probability of catastrophe from full speed’s 75 percent, in percentage points, for each rule at the productivity on the slider. Blue bars lower risk; orange bars raise it.

The implication is that a ceiling and an allocation rule make different promises. One fixes hazard and lets delay vary; the other fixes dates and lets risk vary.

The math

Part 5What a regulator can promise

Step 14 · Paper, Section 9

What a regulator can promise

A regulator can proceed in three steps: establish feasibility, choose compliant timing, and compare the cost to customers. The feasibility test has three inputs: the log reduction in hazard the ceiling requires, ln[h(1)/c̄]; the fastest attainable learning per unit of productivity, σ̄maxȳ; and the time the lead allows, T0 + L. It has three outcomes.

  1. If the necessary condition fails at the productivity bound, no rule keeps both promises. One of them must be relaxed, the safety technology improved, or the final model made less hazardous.
  2. If the replay condition holds, learning first and then replaying keeps both, at the service cost of Step 11.
  3. Between the two, the question is open: the paper’s bounds neither rule out a policy nor provide one.

Then screen the possible pause dates with the three checks of Step 10 and rank the survivors with the comparison of Step 9. Later research can be more productive, but it cannot erase a ceiling violation before learning begins. Finally, evaluate customer services on the compliant paths; the policy with the lowest risk need not have the lowest service cost.

In the scenario, a regulator who cannot rule out productivity of half the fitted value cannot promise the ceiling and the deadline together, by any rule. One who can rule out productivity below 0.69 can promise both with the spare-compute replay, at 13 percent of five-year services. One who can rule out productivity below the fitted value pays only 4.1 percent. Between 0.57 and 0.69, the answer is open. So measuring how much safety research reduces hazard, and how well that knowledge carries to stronger models, is what makes a joint promise affordable.

A monthly ceiling bounds the probability of catastrophe only loosely: with the deadline it implies at most 77.2 percent. The simulated risk is far lower, 8 percent for the spare-compute replay at its bound and 2 percent at the fitted productivity, because the ceiling caps the hazard rather than setting it.

The analysis has limits. Compute grows exogenously, so the model leaves out a lab that invests to restore the speed a research share takes away. The overhang result assumes step hazard proportional to speed. Enforcement and the measurement of hazard are idealized: compute can be detected and training told from inference, but safety research is harder to tell from capability research. The deadline is a fixed limit on delay rather than the outcome of a race, and early knowledge protects later models fully in the main bounds. The scenario’s magnitudes are judgments, not estimates; the analytical results do not depend on them.