The operator-facing error was legacy_runner_failed. It sounded like one runner had failed. In reality, the message came from a projection between two release ledgers, and it no longer revealed whether repository commands had started, which production pointer was authoritative, or whether retrying could repeat a side effect.

Forge did not have one unreliable deployment pipeline. It had two deployment authorities describing the same release. Legacy Sites owned builds and a serving pointer; generalized Deploy owned releases, provider activation, and another production pointer. The compatibility layer tried to reconcile their accounts after the fact.

We deleted that bridge and made generalized Deploy the only normal authority. When the release system later had to deploy its own repair, a clean exact-SHA build and immutable provider upload restored the control plane without moving traffic; final activation returned to generalized Deploy. The result is structural, not a benchmark: one source of truth and one tightly bounded recovery path outside its failure boundary.

A retry can be safe inside one state machine and still worsen an incident when two systems claim ownership. Self-hosting remains survivable only when the normal path is singular and the exceptional path cannot quietly become a second pipeline.

Two ledgers owned one deploy

The original Forge Sites system was coherent. A push to the production branch created a deployment for one 40-character source SHA. A dedicated runner checked out and built that source in a restricted sandbox. Trusted ingest copied static assets into an immutable R2 prefix. Every successful build received an immutable preview, while production was a pointer to one completed deployment. Promotion and rollback changed the pointer without rebuilding.

That model stopped fitting when Forge grew beyond static hosting. Generalized Deploy added projects, environments, environment-neutral releases, provider activations, durable application artifacts, and connected Cloudflare Workers. Forge itself became the first self-hosted target: build an immutable API Worker version from an exact Forge SHA, verify it, then move traffic behind a compare-and-set fence.

The new system did not replace the old one at first. A default Deploy project could still carry legacy_site_id. Generalized Deploy created a release on push but withheld its own build for a linked legacy Site. The Sites runner did the work, and legacy-release-sync copied the result into deploy_releases through legacy_site_deployment_id.

That left one delivery spread across two ledgers:

• Sites owned repository execution, build artifacts, and a serving pointer.
• Deploy owned the intended release, provider activation, and an environment pointer.
• The bridge copied identities, generations, errors, artifacts, and receipts between them.

This was not harmless duplication. The bridge decided which facts became visible and which details disappeared.

Projection hid the failure phase

The clearest symptom was the error code legacy_runner_failed. It named the compatibility boundary, not the failed phase. It did not say whether repository commands had started or whether a durable post-build artifact existed.

That missing distinction controls retry safety. A release interrupted before repository code starts can be replayed automatically. An interruption after arbitrary repository commands begin is ambiguous: the command may have changed an external system before its receipt disappeared. The old retry policy correctly refused to replay that case, but the compatibility projection could not explain why. Operators saw a Sites failure, a Deploy release, and pointers that did not necessarily describe the same outcome.

Authority contraction Two ledgers became one release path

Sites ledger build + Sites pointer

Compatibility sync projected IDs, errors, artifacts

Deploy ledger activation + environment pointer

delete the projection boundary

One deploy_release exact source + stable identity

One generalized build phase-aware interruption evidence

One production pointer moves after exact-version proof

Generalized release runs now persist a stable interruption code, the exact phase, whether repository commands started, and a bounded retry disposition. Reconciliation can repair missing delivery state or redispatch work that never started. It cannot infer permission to rerun an ambiguous repository command, and it cannot manufacture another Action run.

Preproduction made deletion possible

Compatibility is valuable when it protects real users or an established contract. This bridge did neither. Forge Deploy was still preproduction, so preserving both paths created operational cost without preserving a customer promise.

The read-only cutover preflight classified every compatibility row before destructive work. Its saved report contains 807 records: 795 inactive records were disposable, 12 active applications required a fresh generalized build at their exact source SHA, and zero rows blocked the contraction. The classifier also checked whether the currently served provider version could become an immutable rollback baseline. It did not treat an existing row as trustworthy merely because it existed.

Commit f300a0e then made generalized releases the sole authority across 178 files. It added 3,723 lines for the one-way migration, reconciliation, retry, and acceptance contracts while deleting 19,448 lines, including the Sites deployment APIs, legacy runner path, projection model, and serving decisions.

Deletion was the feature. Teaching the bridge another recovery case would have preserved the same question for every future incident: which ledger and which pointer are real?

The self-hosted check called back into itself

Forge was designed to deploy Forge, but not to make Forge its only recovery mechanism. The original self-hosting boundary already kept Cloudflare as the runtime and final break-glass authority. Forge owned the normal exact-SHA release and audit trail; an operator could still inspect or upload an immutable provider version without first making Forge healthy.

The incident showed why that separation matters. The runner verified a staged Forge API candidate through the public API route. The API had a private CI_RUNNER service binding that could route back to the same runner. A Worker-global request therefore crossed from runner to API and back into runner, creating a service cycle. Cloudflare could return 522 even when the candidate version itself was healthy.

Commit af5f53c moved that health canary into an isolated sandbox network path. The canary uses a version override, requires the expected immutable version and binding contract, and waits for three consecutive matching responses. Verification now crosses an independent network boundary instead of depending on the service it is testing.

Bootstrap the owner, then give authority back

Once the release machinery was the blocker, another self-release would add a dependency on the failing system. The escape sequence was intentionally narrow:

clean checkout at an exact SHA
  → build the owning control-plane version locally
  → upload an immutable provider candidate without changing traffic
  → verify source identity, bindings, deep health, and rollback target
  → resume the generalized release
  → move the production pointer only after exact-version hostname proof

The local upload answered one question: can we restore the mechanism that owns releases? It did not answer which source Forge should record as deployed or which version should receive production traffic. The generalized release ledger and fenced environment pointer still owned those decisions.

This is why the article makes no percentage claim. We did not run the same recovery twice under controlled conditions. The incident also included the one-way schema cutover, contract tests, provider verification, and Git canaries. The defensible result is structural: the bootstrap removed the circular dependency without establishing a permanent shadow release path.

Break-glass needs hard limits

A local build and provider upload is appropriate only when the release owner is itself the blocker. We require all of the following:

1. The source checkout is clean and identifies one exact 40-character SHA.
2. The candidate is immutable and can be uploaded without moving current traffic.
3. The existing production version is recorded as an explicit rollback target.
4. Source identity, provider version, required bindings, and deep health are checked independently.
5. No ambiguous schema or application-data mutation is hidden inside the bypass.
6. The bypass restores the control plane; final application delivery returns through generalized Deploy.

The release tooling now encodes the same shape: validate a clean exact source, inspect the uploaded provider version, keep the old 100-percent version as rollback evidence, and report upload, activation, pointer state, and live verification separately.

We will not use this path because a healthy build is inconvenient or because “version uploaded” is an easier stopping point. Those shortcuts would turn recovery into a second deployment authority and recreate the original fault.

Agents must escalate shared failures

The architecture change also needs a behavioral contract. An agent asked to “retry this application deploy” will keep changing that application if the platform presents a plausible repository-local error. That is rational inside the assigned boundary and damaging when the same failure belongs to shared infrastructure.

The escalation signal is correlation. If the same signature survives changes to repository, branch, payload shape, and trigger, stop modifying workloads. Preserve the exact source state, name the common ingest, runner, or control-plane boundary, and repair its owner.

Agents and operators must also report release evidence as separate facts:

• source commit and canonical branch;
• required Action and its conclusion;
• generalized release and build attempt;
• immutable provider version and binding contract;
• active environment pointer; and
• live hostname proof for that exact version.

A green Action is not a deployment. An uploaded version is not active traffic. A healthy hostname is not proof of the requested SHA. Keeping those statements separate is less elegant than one green badge, but it prevents a familiar category error from entering the release ledger.

The durable boundary

The evidence here is specific. The preflight proves the legacy inventory and zero-blocker cutover state. The sole-authority commit proves the active compatibility code was removed. The service-cycle test proves the exact-version health check now runs outside the recursive API-runner path. None of that is a benchmark of local deployment speed, evidence of general availability, or permission to bypass a healthy release system.

The durable lesson is smaller: a self-hosted platform needs one normal deployment authority and one independently reachable recovery boundary. The recovery path should restore the authority, produce the minimum evidence needed to trust it, and then disappear from ordinary use. Forge later applied the same separation to coding work by stopping feature agents from deploying.

The next test is operational, not architectural: exercise that break-glass sequence under a declared failure, measure the recovery time, and prove the bootstrap leaves no unrecorded provider version or pointer change behind. Until then, we know the loop is removable. We do not yet know how quickly every operator can remove it under pressure.