The previous Forge field note said exact-SHA archives cut a large repository's repeat checkout from about two minutes to about 14 seconds. In conversation, that precise result quickly became a simpler memory: we cut the build to 14 seconds. Then Forge's own Platform CI completed a run in almost 16 minutes.

The first question was the obvious infrastructure question:

“Are slow and interruptible builds mainly a capacity problem—vertical scaling or horizontal concurrency—or are there architectural and operational causes […]?”

Production evidence answered neither scaling move first. Queue p95 was about three seconds, observed concurrency peaked at six of ten runners, and the job already used the largest predefined container size. The slow run spent almost 14 minutes inside one serial release gate. Forge made that gate measurable, stopped rebuilding the same web assets, and elected one trusted exact-cache producer. A later successful run finished in about 5½ minutes, including a 4½-minute gate. That later run is a production comparison, not a controlled benchmark; the directly measured cache cohort cut duplicate post-run work about 80%.

Fourteen seconds was one boundary

The earlier result was real. A trusted push published an immutable source archive keyed by repository and exact commit SHA. Repeat consumers restored and verified that archive in about 14 seconds instead of preparing source for about two minutes.

But source checkout is one stage. It does not include dependency installation, typechecking, unit tests, release contracts, production builds, integration tests, smoke tests, audits, artifact publication, or cleanup. The phrase “14-second build” erased all of those boundaries.

Platform CI run 721 made the difference hard to ignore. It waited about two seconds for a runner, restored source in about 11 seconds, missed the dependency cache, installed packages in about 92 seconds, and then spent almost 14 minutes in the complete release gate. Total execution was about 16 minutes.

The run immediately before it exposed the reliability cost. Its first package install hit a network reset and repeated. After about three minutes of dependency work, Cloudflare Workflows returned an internal error while repository commands were already running. Forge correctly refused to replay an arbitrary command with unknown side effects. The explicit same-SHA rerun started from the beginning.

Fast checkout had removed one expensive stage. It had not shortened the serial proof that followed it, made cache publication reliable, or created a safe checkpoint after installation.

Two clocks answered the scaling question

Horizontal scaling helps when work waits for a runner. Forge's queue did not.

Across 40 Platform CI runs, queue p50 was about two seconds and p95 was about three. At most six container jobs were active against a configured ceiling of ten. The 16-minute rerun executed alone. Adding more runners would have changed almost none of its wall time.

Vertical scaling was not the evidence-backed answer either. The runner already used standard-4: four vCPUs, 12 GiB of memory, and 20 GB of disk. More important, Forge did not yet record enough per-stage CPU, memory, and disk evidence to identify a resource ceiling. There was no out-of-memory event, disk exhaustion, CPU-limit error, or timeout in the interrupted run.

This left a third explanation: the runner was doing too much serial and repeated work. Scaling could conceal that work under more hardware or improve unrelated queue pressure, but it would not remove the work or make an interrupted command safe to replay.

Forge therefore added two clocks instead of adding capacity. Durable attempt records now heartbeat independently from command output and preserve the current phase, last step, runner generation, sandbox identity, resource counters, and structured exit class. The release gate separately records the duration and resource delta of each proof stage.

That distinction matters. A quiet command is no longer automatically indistinguishable from a dead runner, and a slow gate is no longer one 800-second black box.

Instrument first, then delete repetition

The original release command was a long shell chain. It typechecked every workspace, ran the package suites, ran contracts, built every buildable workspace, started two local smoke environments, and audited production dependencies. Several stages compiled the same code again.

The first change did not parallelize the gate. It made ten named stages visible and kept them serial, so timing changes could be attributed. Then Forge removed work whose proof already existed:

• Full typechecking remained authoritative, but the later web asset build stopped running TypeScript again.
• The gate built the exact web Worker asset tree once.
• Both local smoke tests verified and reused that stamped asset tree instead of rebuilding it independently.
• Contracts that belonged to retired preproduction or migration paths left the active release gate after the one-way cutover completed.

The later successful gate reported about 53 seconds for typechecking, 24 seconds for API tests, 23 seconds for Worker tests, 35 seconds for product tests, 89 seconds for contracts, 22 seconds for the production asset build, and about 11 seconds for each smoke path. The complete gate took about 4½ minutes.

Those numbers now identify the next targets. Contracts and typechecking dominate the remaining serial work. More runners still will not shorten them unless Forge first turns independent proof into separate jobs or shards with explicit inputs and durable results.

One cache producer was enough

One push can start Platform CI, Git ingest, and a Site build for the same SHA. Before coordination, each successful consumer could reach the end, discover the same missing exact dependency snapshot, and compress the same 233 MB node_modules tree.

The first single-producer attempt waited 60 seconds. That sounded generous. Production proved otherwise: one Action exhausted the wait, spent another 58 seconds preparing the snapshot, and completed about 129 seconds of post-run cache work while a Site prepared the same immutable payload.

Forge increased the bounded wait to 90 seconds using the observed 58–65-second preparation and 10–17-second upload window. A same-SHA Site is preferred because its provider ingest already depends on the Forge-owned frozen install. Otherwise the earliest eligible Action publishes. Siblings poll metadata, exit early when a producer succeeds or becomes terminal, and retain a post-preparation recheck to close the race.

trusted Site or earliest Action
  └─ frozen install → prepare once → publish repo + SHA + toolchain fingerprint

sibling Action
  └─ wait for immutable row → already published → skip preparation and upload

The next production cohort prepared the snapshot once in about 45 seconds and uploaded it in about 11. A sibling observed the immutable row after about 25 seconds and skipped both operations. Its post-run cache work fell from about 129 seconds to about 25 seconds—roughly an 80% reduction.

On the later full Forge run, the exact-cache save check took 34 milliseconds and reported already_published. Untrusted pull-request builds may consume a matching exact snapshot, but they remain read-only. The faster path did not add an expensive validation pass after arbitrary repository code; it narrowed write authority to a trusted, pre-build artifact boundary.

The later run was five and a half minutes

The first successful Platform CI run after the complete gate and cache changes finished in about 5½ minutes. Its measured gate took 4½ minutes, down from almost 14 minutes in the original slow run. Total execution was about 65% lower.

That comparison is useful operational evidence, but it is not a controlled benchmark. The runs used different commits. The later run restored a manager cache; the earlier run missed. Several release contracts and smoke paths changed between them. Production load also differed.

The causal claims are narrower:

• queue evidence ruled out horizontal capacity as the cause of the 16-minute run;
• missing resource evidence ruled out a confident vertical-scaling prescription;
• code inspection proved that compilation and asset builds repeated inside the serial gate;
• stage telemetry measured the later gate at 4½ minutes and exposed its remaining distribution;
• the cache cohort directly measured one-producer coordination cutting duplicate post-run work about 80%.

That is enough to choose the next engineering move without pretending one production pair is a percentile study.

Faster is not yet resumable

The runner reliability work also added renewable attempt leases, 30-second heartbeats, a 90-second expiry, attempt fencing, structured exit evidence, and enforced draining before runner releases. Those changes make interruption diagnosable and prevent a rollout from starting while Action runs remain active.

They do not make arbitrary repository commands resumable. The entire container job still has coarse durable boundaries. If infrastructure disappears after a side-effecting command begins, Forge continues to fail closed and requires an explicit exact-SHA rerun.

Safe resumption needs stage checkpoints whose inputs include the exact SHA, toolchain, lockfile, and relevant configuration. A completed deterministic stage may be reused. An in-progress command with unknown side effects may not. That work is separate from adding runners and separate from making one cache upload survive a transient HTTP 520.

Optimize the measured boundary

The durable lesson is not “never scale.” Horizontal capacity is the right response when queue age grows under saturation. Vertical capacity is the right response when stage-level resource evidence shows CPU, memory, or disk pressure. Neither condition described this incident.

Forge had optimized checkout and remembered the result as a build result. The 16-minute run forced the system boundary back into view. Once queue time, gate stages, and cache publication were measured separately, the first cuts were straightforward: compile the asset once, reuse it in both smokes, and publish one immutable dependency snapshot per exact input.

The next cut should follow the same rule. Contracts and typechecking are now the longest measured gate stages. Forge can split or parallelize them only after their inputs and outputs become independent, durable evidence—not because another runner happens to be available. The same distinction between fast iteration and complete proof shaped Forge's test-architecture cleanup.

The raw cache measurements and trust boundary are recorded in the runner source and dependency caching decision. The implementation landed through the measured release gate, single-build smoke reuse, and bounded single-producer cache wait.