Forge Deploy trial: reliability findings and hardening backlog#

Status: operational notes from the first end-to-end sites.smol.ai fixture trial. This is not a production-readiness claim. It records what happened when Forge accepted real Git pushes, built and published the three fixtures, and served them from the hosted edge.

Executive summary#

The trial proved the intended end-to-end shape works:

Git push -> exact SHA -> build -> trusted ingest -> immutable deployment -> hosted URL.

It also exposed a reliability gap common to multi-service platforms: a Git push can be accepted while later deployment work is silently absent or blocked by a live-version/configuration mismatch. The fixes below are about making that situation observable, recoverable, and eventually impossible.

The important product rule is:

A successful Git ref update is not deployment success. Forge must show the handoff, build, ingest, provider publication, activation, and live-hostname evidence as separate states.

Shipped reliability slice#

The first four contracts were rolled out and verified on their owning Cloudflare surfaces on 2026-07-24.

Verification covers fresh and upgraded schema application, binding-manifest drift, TypeScript contracts, retry/backoff policy, diagnostic redaction, runner identity admission, and the existing API/runner/edge/runtime behavioral suites.

Phase 0–2 live conformance#

• Backfill completed for three projects, 41 releases, and three legacy activation snapshots. A selected static projection was equivalent.
• Ten concurrent reservation claims produced exactly one success and nine conflicts.
• Production reservation state remained present across code rollback and restoration; the immutable preview had independent empty state.
• Two realtime clients received one mutation event. The first 20 preview connections succeeded and connection 21 received terminal 429 realtime_quota.
• An expired synthetic artifact was deleted from R2 while the deployment, cleanup operation, and lifecycle events remained queryable.
• The source-visibility lifecycle, noindex, automatic publication protection, and rollback reconciliation soak passed on live hostnames.

The pre-migration recovery set includes a D1 Time Travel bookmark, a full SQL export, and a schema-only export. The schema-only export imported into scratch SQLite with integrity_check=ok. The full export downloaded and was hashed but is not represented as restore-tested because a transcript data statement did not parse in scratch SQLite.

Observed incidents#

Reliability contract to add#

1. Make the Git handoff durable#

The immediate waitUntil() change prevents a streaming-response timing race, but it is not the final reliability boundary. A correct control plane needs a database-backed outbox.

1. Update the ref and insert a repository_ref_changed outbox record in one transaction.
2. A durable dispatcher reads the outbox, creates the idempotent deployment record, and marks the event delivered only after the handoff commits.
3. A reconciler scans enabled Sites for a production-branch head with no matching deployment record after a short bounded delay.
4. The reconciler repairs the handoff using the same idempotency key and opens an operator alert if it cannot.

The idempotency key should include repository, ref, exact SHA, trigger type, and effective project configuration version. Replays must create at most one deployment for the same intended work.

2. Treat missing hosting configuration as an outage#

An absent hosting binding is not “this repository has no Site” once Forge has already identified an enabled, eligible Site. The trigger should:

• return normally only when no Site is eligible for that ref;
• emit a structured, redacted configuration error when an eligible Site cannot be scheduled because a required binding, provider namespace, or service is absent;
• increment a deployment-handoff failure counter and show the error on the repository’s Deploy page;
• page or create an operator incident when the error is global rather than project-specific.

At deploy time, each Worker must validate its required bindings and variables against a checked-in release manifest. The deployment should fail before traffic is switched if the actual binding set differs from that manifest.

3. Version the whole execution tuple#

Every deployment should record the versions that actually executed it:

• Forge API script version and configuration digest;
• runner Worker script version;
• runner container image digest;
• sandbox/runtime library version;
• Forge-owned toolchain manifest, including esbuild 0.25.8, Wrangler 4.113.0, SQLite 3.37.2, each binary path/digest, and one aggregate contract digest;
• artifact-ingest policy version;
• edge Worker version;
• provider adapter and dispatch namespace version.

The UI should display these in an advanced “execution evidence” panel, while support tooling can search them directly. A deployment should only start when the runner has reported the requested image/toolchain as healthy.

4. Make runner upgrades safe#

Deploying a Worker that owns Durable Objects or container instances is a rolling operational event, not merely a code upload.

Provider traffic can point at the new Worker before every Sandbox reservation observes the new container image. --containers-rollout=immediate does not make those two completion proofs atomic. Release verification must keep the exact Worker version pinned while it retries only explicit toolchain-convergence codes. Forge bounds this to six probes with 110 seconds of total delay and re-publishes verifying before each probe because a failed probe moves control to failed. Any different Worker version, non-toolchain binding failure, or exhausted budget fails the release and enters the normal rollback path.

The image cannot define which digest an arbitrary Worker version validates. Current Workers publish readiness.observed_toolchain_digest; Deploy control persists that exact value. Compatibility with an older Worker is a field- absence fallback to its checked esbuild digest, including when that Worker sees a v2 image. If a Worker publishes the field with an invalid value, verification fails closed rather than silently selecting a digest from image metadata.

Required upgrade behavior:

• mark the runner as draining before the rollout;
• stop assigning new jobs until a health probe reports the new Worker, container image, fixed toolchain, R2 binding, and runtime-service binding;
• let in-flight jobs finish or checkpoint them;
• retry known transient reset/restart errors with exponential backoff and an idempotent run token;
• publish ready only after a real synthetic static build and dynamic bundle check pass;
• route failed post-upgrade jobs to retry rather than immediately labelling their user deployment terminally failed.

The hard-coded esbuild executable path should become a checked startup assertion. Better still, the trusted toolchain should expose one Forge-owned wrapper command whose version and digest are recorded; bundling code should not rely on an undocumented global path.

5. Validate the effective build plan early#

Deployment UI and API should show, for the exact commit:

Before allocating a runner, Forge should reject obvious contradictions:

• a nonempty build command requiring a package manager but no matching project file at that SHA;
• a declared output root missing after an explicit no-build path;
• an application entrypoint absent or non-regular;
• manifest assets/fallback paths that are not present after build;
• configuration changes made after the deployment was queued.

This turns a remote runner failure into a clear configuration diagnosis.

6. Harden trusted ingest and artifact publication#

The fixed-length upload requirement is correct: the trusted ingest path should know the byte count before R2 accepts a stream. Keep these invariants:

• quiesce the build sandbox before trusted reads;
• verify each regular file’s observed size and digest after quiescence;
• use an explicit fixed-length stream (or an equivalent verified byte buffer) for every object upload;
• await both source-to-upload piping and R2 completion;
• verify returned object size and record ETag/digest in the manifest;
• write the completed artifact manifest only after every object is present;
• make retries reuse deterministic prefixes and safely overwrite or verify the same object identity.

Add a remote integration test that uploads a streamed artifact through the actual Worker binding. Local mocks alone cannot detect provider stream semantics.

7. Separate and reconcile lifecycle states#

Forge should explicitly present these independent facts:

stateDiagram-v2
  [*] --> RefUpdated
  RefUpdated --> HandoffQueued
  HandoffQueued --> DeploymentCreated
  DeploymentCreated --> Building
  Building --> Ingesting
  Ingesting --> ProviderPublished
  ProviderPublished --> Verified
  Verified --> PreviewReady
  PreviewReady --> Active: auto publish or promote
  Building --> Failed
  Ingesting --> Failed
  ProviderPublished --> Failed
  Failed --> Retrying
  Retrying --> Building

For each edge, retain a timestamp, attempt number, idempotency key, and a sanitized diagnostic. The UI must never collapse RefUpdated into PreviewReady or Active.

Observability and operator experience#

One deployment evidence record#

For every deployment, generate a compact support bundle containing:

• repository, branch/ref, exact SHA, actor, and Git receive request ID;
• outbox event and handoff attempt history;
• deployment/workflow/job/step IDs and current states;
• configuration, manifest, plan, and policy digests;
• API, runner, edge, image, and provider version identities;
• artifact prefix, file count, bytes, and manifest digest;
• preview and production endpoint status checks;
• runtime script name/version when dynamic code exists;
• quota counters and suspension/access epochs;
• sanitized errors, retry classification, and next operator action.

Provider identifiers belong in this advanced evidence record; Forge IDs remain the primary product handles.

Metrics and alerts#

Track at least:

• git_ref_to_deployment_created_seconds and the count of eligible refs with no deployment after the service-level objective;
• deployment success rate and terminal failure rate by phase;
• runner image/toolchain mismatch count;
• Durable Object reset/restart retry count;
• artifact ingest size/stream/manifest failures;
• provider publication and activation latency;
• live endpoint smoke success, separately for static, HTTP, state, and realtime fixtures;
• retry attempts, retry exhaustion, and reconciler repairs.

Alerts should distinguish a single repository configuration error from a global API-binding, runner-image, R2, dispatch, or edge outage.

Usage summary and traffic dataset#

GET /api/admin/deploy-reliability/usage is restricted to platform Site administrators. It reports the durable D1 control-plane totals for repository event and deployment queues, successful deployment artifact files/bytes, and deployment counts and reuse totals by execution_path.

Request and bandwidth usage has a different owning surface and is not copied into D1. The Sites edge writes one point per response to the Cloudflare Analytics Engine dataset forge_sites_usage_preview through the SITE_USAGE binding:

Cost and bandwidth reporting must query this dataset by Site and time window. The D1 endpoint intentionally returns the dataset contract rather than presenting incomplete request or byte totals as authoritative.

Security and abuse boundaries#

Reliability machinery can itself become an abuse multiplier, so the trial slice applies the following limits:

• Ref triggers create outbox work only for enabled Sites. Ordinary repository pushes do not consume Deploy outbox storage.
• Dispatch and reconciliation batches are capped at 25 records per pass. Leases expire after 60 seconds; retries stop after eight attempts and use exponential backoff capped at 15 minutes.
• Dead-letter events are retained for diagnosis rather than retried forever. The operator view reports backlog and incident counts.
• Git-receive annotations consider at most 100 branch changes per request.
• Operational messages are length-bounded and redact bearer tokens, common secret assignments, and URL credentials before persistence or logging.
• Runner state and release controls require an authenticated Site operator. A direct transition to ready is forbidden; readiness requires the deep runner probe.
• Deployment evidence is visible only to repository administrators because it contains internal version and provider identifiers. Public hosted output does not expose it.
• Health endpoints disclose contract/version state but never secrets. Missing required bindings fail closed.

Before broader self-service, add retention for delivered/dead-letter outbox rows, per-repository handoff rate metrics, anomaly alerts for repeated ref churn, and a global emergency pause that halts dispatcher claims without deleting intent. These are capacity and abuse controls, not substitutes for existing repository authorization or Sites suspension.

Staged verification plan#

Run these in order so a failure is attributable and rollback remains small:

1. Database contract: apply all five canonical fresh-schema fragments to an empty local D1 database, verify foreign keys/triggers/default runner state, then update one enabled project ref and confirm exactly one pending event is produced. No upgrade or ledger-adoption path exists.
2. Control-plane contract: deploy the API with its full binding manifest, inspect /health, run one normal push, and verify the ref event, deployment, workflow, and handoff timeline share the expected SHA.
3. Recovery contract: temporarily block runner admission, push twice, and prove intents remain pending with bounded retries. Restore readiness and prove each intended SHA creates at most one deployment.
4. Release contract: upload—but do not activate—a runner version/image, set draining, wait for active work to settle, set verifying, and invoke the deep probe. Only a matching Worker/image/toolchain tuple may become ready.
5. Execution-evidence contract: deploy one fixture and compare the evidence panel with Cloudflare Worker versions, the runner image digest, dispatch namespace, deployment record, and live hostname.
6. Chaos contract: repeat with a Git disconnect, an expired dispatcher lease, a runner reset during setup, a reset after repository code starts, an R2 interruption, and a missing API binding. Confirm safe replay where possible and an explicit blocked/terminal state everywhere else.
7. Abuse contract: generate rapid ref churn for one allowlisted fixture, verify batch/retry caps and no duplicate deployments, then suspend the Site and confirm new intent is not created or dispatched.
8. Conformance contract: rerun static, reservations, and realtime fixtures; verify SPA fallback, exactly-one reservation claims, state persistence across code rollback, realtime resync/quota refusal, egress denial, and immediate disablement.

Fresh D1 schema#

Forge has no production data contract yet. Schema changes replace the five size-bounded fragments of one fresh-install schema and require recreation of the pre-production D1 database. The verifier applies those fragments to an empty database and fails on integrity, foreign-key, required-object, seed-signature, or forbidden compatibility-object mismatches. No ledger-adoption or upgrade procedure exists.

Acceptance and chaos checks#

Before opening the trial beyond the allowlist, automate these remote checks:

1. Push each fixture and prove a deployment record appears within the handoff SLO, even if the Git client disconnects immediately after refs update.
2. Deploy an API version missing a required hosting binding in a staging environment; prove an eligible Site creates an observable configuration incident rather than silently doing nothing.
3. Roll the runner Worker and container image during queued fixture builds; prove jobs drain or retry and no run is abandoned.
4. Verify a runner startup probe finds the pinned esbuild wrapper and performs a minimal bundle before dynamic jobs become eligible.
5. Exercise streamed asset uploads against real R2 with zero-byte, small, multi-megabyte, and interrupted files; verify retry idempotence.
6. Configure an assets-only fixture with no build command; prove it deploys. Configure a stale command; prove Forge rejects it before runner allocation.
7. Confirm static SPA navigation uses an HTML navigation request and falls back to index.html, while an asset fetch still returns a real 404.
8. For reservations, send concurrent claims and prove exactly one succeeds; roll back code and prove production durable state persists.
9. For realtime, open multiple tabs, verify update/resync behavior, quota refusal, suspension, and reconnect after a runtime rollout.
10. Disable a Site and prove edge serving, runtime calls, tickets, mutations, and sockets stop immediately.

Prioritized backlog#

P0 — before more trial users#

• Database outbox and reconciler for ref-to-deployment handoff.
• Required-binding release manifest and deployment-time verification (local; staged activation rehearsal pending).
• Fail-loud trigger behavior for eligible Sites with missing hosting configuration.
• Runner rollout drain/health gate and retry classification for code-reset errors.
• Record API, runner, image, and edge versions on every new deployment.
• One repository-admin deployment evidence view in the UI.
• Persist live-hostname smoke checks in the deployment evidence record.

P1 — before self-service expansion#

• Effective-build-plan preview and preflight validation.
• Pinned, self-verifying Forge toolchain wrapper; remove reliance on an opaque global executable path.
• Remote integration exercise covering R2 cleanup, dynamic dispatch, runtime state, realtime, disablement, and rollback.
• Phase-specific retry policy, retry budget, and actionable surfaced errors.
• Deployment SLOs, dashboards, and global-versus-project alert routing.

P2 — production graduation#

• Separate registrable hosting domain, TLS/DNS rollout and rollback rehearsal.
• Multi-provider adapter health/evidence model before Railway or Modal workloads are enabled.
• Automated preview retention and artifact garbage collection.
• Full cost attribution and abuse operations.
• Full disaster-recovery exercise for control-plane records, immutable artifacts, and runtime state boundaries.

Session takeaways#

1. The overall architecture is viable, but availability comes from the handoffs between Forge services, not from any single Worker succeeding.
2. Exact-SHA execution is necessary but insufficient: the exact control-plane configuration, code version, image, toolchain, and provider state must be attributable too.
3. “Retry” is a valid recovery tool only when it is deliberate, idempotent, classified, and visible. Manual empty commits are acceptable during a tightly controlled trial, not as a product mechanism.
4. The three fixtures are valuable as permanent conformance probes. Keep them deployed and run them after every control-plane, runner, edge, or provider rollout.