Forge release state machine v2#

This is the normative state and authority contract for platform releases. The machine-readable policy is ../../config/forge-release-system-v2.json. The incident basis is ../operations/release-modernization-incident-synthesis.md.

Authority and durable truth#

An immutable release record and required-component manifest in D1 are the durable read model. Provider versions, artifact digests, traffic, bindings, triggers, and hostname observations are external facts referenced by that record. An environment-scoped Durable Object serializes managed-Worker promotion and owns its monotonic fencing token independently of D1. Connected edge promotion is serialized separately by generalized Deploy's provider-target lease, exact provider-version compare-and-swap, and production-pointer generation.

The Durable Object is not a second edge deployer. Losing, expiring, cancelling, or superseding its managed-Worker lease makes all earlier managed-Worker tokens permanently stale. The controller enqueues an exact edge artifact and observes its generalized Deploy receipt; it never performs the edge traffic mutation.

Only generalized Deploy may activate connected edge artifacts. Direct local provider access is audited break-glass recovery only and may never compete for traffic, triggers, or active pointers.

Snapshot isolation#

Each release is identified by its D1 release ID, exact source SHA, exact base SHA, and sealed plan digest. The controller must seal that plan before entering promoting, and every provider authorization revalidates the digest. The current branch head is never an input to resume, retry, activation, or rollback.

A later main update only enqueues another release. Descendants may coalesce while queued or validating; after provider work begins, the active release finishes or rolls back under its existing fence before a later release may promote. These rules are executable policy in releaseIsolation inside the machine-readable configuration.

Canonical phases#

resolve_source
→ authorize_environment
→ acquire_lease
→ read_provider_provenance
→ calculate_impact
→ resolve_validation_evidence
→ run_validation
→ resolve_artifacts
→ verify_bootstrap
→ promote_components
→ reconcile_edge
→ run_production_canary
→ verify_public_manifest
→ mark_deployed
→ enqueue_large_snapshot
→ publish_receipts
→ release_lease

Phases are monotonic for one fenced lease. Resumption reads the durable release record and provider facts, rechecks the fence, and continues at the first phase whose completion receipt is missing or no longer true. It does not infer completion from a lost command response.

Release states#

Terminal states are deployed, degraded, failed, cancelled, and superseded, plus rolled_back. A release never transitions directly from build or promotion to deployed.

Required-component manifest#

The final manifest is forge-required-component-manifest/v2. It contains every required component, including unaffected components legitimately running an older source SHA. Each component entry identifies:

• source SHA;
• immutable artifact digest;
• provider version ID;
• binding-policy/contract digest;
• expected traffic;
• rollback provider version;
• dependency-health proof;
• public or service-binding observation as applicable.

The planner initially emits a draft. A selected component is immutable-artifact-required; an unaffected component may reuse-live-provider-version; missing live provenance remains unresolved and fails closed. The manifest becomes immutable only after every selected artifact and expected provider version has been resolved, before promotion begins.

deployed requires all of the following:

• current managed-Worker fencing token, or the connected edge provider-target lease and exact production-pointer generation;
• exact required-component manifest;
• exact provider versions and unambiguous traffic;
• required bindings and dependency graph healthy;
• edge version, triggers, and traffic reconciled;
• correct version on every public hostname;
• successful small push, clone, and strict-fsck canary.

Optimized large-snapshot readiness is deliberately absent from this list. It is an asynchronous readiness sub-state with its own receipts and SLOs.

Large-snapshot handoff and attempt history#

enqueue_large_snapshot means only that the coordinator durably submitted one job and persisted repository_id, refs_generation, and job_id in release evidence. Snapshot completion is not a lease phase and cannot delay deployed. After durable enqueue and synchronous receipt publication, the coordinator releases the production lease. A separate reconciler observes queued, running, ready, failed, cancelled, or superseded readiness.

Every execution attempt is append-only. A terminal attempt is never reset to queued or running. A retryable failure creates the next numbered attempt on the same job. Operator requeue creates a new job ID with operator identity, reason, and prior job ID. Generation supersession also creates a new job ID, names the prior terminal job, and requires a strictly newer refs generation. These rules preserve exhausted-attempt evidence while allowing safe recovery.

The 30-second snapshot SLO applies to a warm incremental generation after a small ref change. It is not a cold-snapshot deadline. Cold Forge-sized work has a measured speedup target and remains asynchronous even when a repository has thousands of objects or hundreds of megabytes of pack data.

Planning and evidence#

Affected CI and the controller read the exact Git tree, hash every declared component input, expand workspace dependency verification, and emit forge-release-plan/v2. The controller compares those digests with successful per-component activation receipts; provider source SHAs remain provenance and are not impact baselines.

Ordinary changes validate affected components. Schema, authority, binding, shared-contract, release-system, explicit-full, or uncertain changes force the documented broader policy. Release-system-only changes rerun contract proof without rebuilding unaffected Workers.

Reusable feature evidence is forge-feature-ci-evidence/v2, authenticated with HMAC-SHA256 by the trusted runner. Its signature covers the exact candidate, expected target-base SHA, immutable changed files, component input digests, runner identity, result, and expiry. Broader offline validation evidence may use forge-validation-evidence/v2 with Ed25519. Evidence is reusable only when the declared inputs and policy identity match exactly.

The coordinator always reruns source provenance, lease checks, impact planning, recovery preflight, provider authority, binding verification, live health, activation fencing, public manifest proof, and the production canary.

Promotion and failure behavior#

Builds may run concurrently. Promotion order is dependency constrained:

1. canonical D1 schema (verification in normal releases; recreation is never implicit);
2. minimum operator identity bootstrap;
3. deploy-control bootstrap readiness;
4. runner;
5. repository services;
6. supporting domain Workers;
7. edge bindings and triggers;
8. edge traffic;
9. public hostname verification.

A failed required component stops its dependents. Independently successful and healthy promotions remain recorded, the release becomes degraded, final edge activation is withheld, rollback targets remain intact, and the receipt names the exact mismatch plus next_safe_action. The system does not claim a distributed atomic rollback.

Retry is allowed only for named bounded classifications in the v2 policy. Structured application failures, foreign or unexpected bindings, invalid auth, ambiguous traffic, permanently wrong versions, and absent rollback evidence are terminal. Arbitrary sleeps are not part of the contract.

Snapshot R2 transport, Sandbox transport, tar hydration transport, and upload transport failures are retryable within their bounded budgets. Missing Git objects, digest mismatches, and invalid or corrupt packs are integrity failures and terminal. In particular, SnapshotIntegrityError: Git object is missing must never be converted into a generic retryable infrastructure failure.

D1 safety boundary#

Normal release execution is non-destructive. It may verify and apply an approved canonical migration path, but it may not recreate production D1. A D1 cutover is a separate explicit command and approval boundary with export, Time Travel, old/new database IDs, binding verification, bootstrap, authentication, Git reachability, abort, and rollback evidence.

Recovery preflight spans both storage systems. It records the old and new D1 repository identities and their corresponding R2 object namespaces, verifies every restored ref tip and reachable object in the new namespace, and reports missing-object counts before and after repair. Any repair copies only immutable content-addressed objects, verifies every copied object against its content address, and exact-compares restored ref-tip objects. Restoring D1 rows without reconciling R2 namespace provenance cannot pass preflight.

Active release paths#

The protected merge queue, evidence verifier, Durable Object lease, D1 ledger, managed-component executor, and generalized Deploy edge promoter are active. There is one promoter for each component and no mutable runtime switch between competing deployers. Release tasks submit immutable candidates and observe sf release status/watch; they do not negotiate deployment ownership with one another.