# Forge Deploy trial: reliability findings and hardening backlog

Status: operational notes from the first end-to-end `sites.smol.ai` fixture
trial. This is not a production-readiness claim. It records what happened
when Forge accepted real Git pushes, built and published the three fixtures,
and served them from the hosted edge.

## Executive summary

The trial proved the intended end-to-end shape works:

`Git push -> exact SHA -> build -> trusted ingest -> immutable deployment -> hosted URL`.

It also exposed a reliability gap common to multi-service platforms: a Git
push can be accepted while later deployment work is silently absent or blocked
by a live-version/configuration mismatch. The fixes below are about making
that situation observable, recoverable, and eventually impossible.

The important product rule is:

> A successful Git ref update is not deployment success. Forge must show the
> handoff, build, ingest, provider publication, activation, and live-hostname
> evidence as separate states.

## Shipped reliability slice

The first four contracts were rolled out and verified on their owning
Cloudflare surfaces on 2026-07-24.

| Contract | Shipped implementation | Live proof |
| --- | --- | --- |
| Durable Git handoff | Enabled-Site ref mutations create a D1 outbox event in the same SQLite statement. A leased dispatcher creates idempotent deployments; a scheduled reconciler repairs missing handoffs and expired leases. | Migrations are ledgered remotely; fixture pushes converged on one deployment. A rollback soak exposed and fixed a reconciler race that could otherwise rebuild the branch head while older code was intentionally active. |
| Versioned release readiness | Each service has a checked-in required-binding contract, version metadata, and a fail-closed health response. The API records release incidents instead of treating missing hosting configuration as a no-op. | API, runner, edge, and runtime binding digests matched their release contracts before activation. |
| Execution evidence | Deployments receive an append-only phase timeline and an execution tuple containing API, runner, image, toolchain, ingest, edge, runtime, provider, and namespace identities. Repository admins can expand “Execution evidence” in deployment history. | Worker versions, runner image/toolchain, D1 records, dynamic runtime behavior, wildcard hostname responses, and usage counters were checked independently. |
| Runner rollout safety | An operator-controlled `draining -> verifying -> ready` gate blocks new admissions, pins the intended Worker/image identity, and requires a deep toolchain probe. The image exposes a Forge-owned toolchain wrapper with checked esbuild, Wrangler, and SQLite versions plus one digest over the complete contract. Existing Workflow retry classification remains bounded and idempotent. | A runner can become ready only after a deep service-binding probe confirms the pinned image, esbuild 0.25.8, Wrangler 4.113.0, SQLite 3.37.2, and the aggregate toolchain digest. |

Verification covers fresh and upgraded schema application,
binding-manifest drift, TypeScript contracts, retry/backoff policy, diagnostic
redaction, runner identity admission, and the existing API/runner/edge/runtime
behavioral suites.

### Phase 0–2 live conformance

- Backfill completed for three projects, 41 releases, and three legacy
  activation snapshots. A selected static projection was equivalent.
- Ten concurrent reservation claims produced exactly one success and nine
  conflicts.
- Production reservation state remained present across code rollback and
  restoration; the immutable preview had independent empty state.
- Two realtime clients received one mutation event. The first 20 preview
  connections succeeded and connection 21 received terminal `429
  realtime_quota`.
- An expired synthetic artifact was deleted from R2 while the deployment,
  cleanup operation, and lifecycle events remained queryable.
- The source-visibility lifecycle, `noindex`, automatic publication protection,
  and rollback reconciliation soak passed on live hostnames.

The pre-migration recovery set includes a D1 Time Travel bookmark, a full SQL
export, and a schema-only export. The schema-only export imported into scratch
SQLite with `integrity_check=ok`. The full export downloaded and was hashed but
is not represented as restore-tested because a transcript data statement did
not parse in scratch SQLite.

## Observed incidents

| Area | What happened | User-visible effect | Immediate recovery | Durable improvement |
| --- | --- | --- | --- | --- |
| Git-to-deploy handoff | Git receive registered its background work from a pull-driven response stream. The worker could return before `waitUntil()` retained the work. | Branch head moved; no deployment row appeared. | Register a deferred background handoff before returning the streaming response. | Transactional outbox plus a reconciler; alert when an eligible ref lacks a deployment. |
| API release configuration | A live API version omitted `SITES_HOST_SUFFIX`, `DEPLOY_RUNTIME`, and admin bindings. The Sites trigger treated a missing suffix as a no-op. | Git push looked healthy but no Site deployment was scheduled. | Redeploy from the canonical Worker configuration. | Release-time binding contract check; fail closed and log an error for eligible Sites when hosting bindings are absent. |
| Site build configuration | The assets-only static fixture still had `npm run build` recorded in its Site configuration, although its repository intentionally contains prebuilt `dist/` assets. | The deployment failed before ingest because there was no `package.json`. | Clear the stale build command. | Show the effective build plan in the UI before queueing; validate it against the exact SHA; provide an explicit “no build” option. |
| Trusted asset ingest | The runner code used a `FixedLengthStream` to upload R2 assets, but the deployed runner was older than that code. | Ingest failed with a readable-stream-length error. | Deploy the tested runner version. | Pin runner code, container image, and ingest-policy version in every deployment record and verify them before queueing. |
| Runner upgrade lifecycle | The first jobs after a Durable Object/container runner rollout hit “Durable Object reset because its code was updated.” | A short burst of builds failed immediately after the rollout. | Retry after the runner stabilized. | Drain/health-gate upgrades; classify this error as retryable; do not expose a raw terminal failure until retry budget is exhausted. |
| Runner image availability | Dynamic fixture bundling initially reported `/usr/local/bin/esbuild` missing despite the desired image containing it. Existing runner/container instances were still converging. | Dynamic reservations and realtime builds failed; static succeeded. | Retry after the new runner image became active. | Startup self-test of the exact runner image; versioned tool manifest; admission gate that blocks dynamic jobs until the healthy image revision is confirmed. |
| Evidence gathering | Deployment state, Worker version/bindings, runner logs, D1 records, and live hostname each answered different questions. | It was easy to mistake “push succeeded” for “deployment succeeded.” | Trace each surface separately. | One deployment timeline that links every controlling ID and surface, with a single support bundle. |

## Reliability contract to add

### 1. Make the Git handoff durable

The immediate `waitUntil()` change prevents a streaming-response timing race,
but it is not the final reliability boundary. A correct control plane needs a
database-backed outbox.

1. Update the ref and insert a `repository_ref_changed` outbox record in one
   transaction.
2. A durable dispatcher reads the outbox, creates the idempotent deployment
   record, and marks the event delivered only after the handoff commits.
3. A reconciler scans enabled Sites for a production-branch head with no
   matching deployment record after a short bounded delay.
4. The reconciler repairs the handoff using the same idempotency key and opens
   an operator alert if it cannot.

The idempotency key should include repository, ref, exact SHA, trigger type,
and effective project configuration version. Replays must create at most one
deployment for the same intended work.

### 2. Treat missing hosting configuration as an outage

An absent hosting binding is not “this repository has no Site” once Forge has
already identified an enabled, eligible Site. The trigger should:

- return normally only when no Site is eligible for that ref;
- emit a structured, redacted configuration error when an eligible Site cannot
  be scheduled because a required binding, provider namespace, or service is
  absent;
- increment a deployment-handoff failure counter and show the error on the
  repository’s Deploy page;
- page or create an operator incident when the error is global rather than
  project-specific.

At deploy time, each Worker must validate its required bindings and variables
against a checked-in release manifest. The deployment should fail before
traffic is switched if the actual binding set differs from that manifest.

### 3. Version the whole execution tuple

Every deployment should record the versions that actually executed it:

- Forge API script version and configuration digest;
- runner Worker script version;
- runner container image digest;
- sandbox/runtime library version;
- Forge-owned toolchain manifest, including esbuild 0.25.8, Wrangler 4.113.0,
  SQLite 3.37.2, each binary path/digest, and one aggregate contract digest;
- artifact-ingest policy version;
- edge Worker version;
- provider adapter and dispatch namespace version.

The UI should display these in an advanced “execution evidence” panel, while
support tooling can search them directly. A deployment should only start when
the runner has reported the requested image/toolchain as healthy.

### 4. Make runner upgrades safe

Deploying a Worker that owns Durable Objects or container instances is a
rolling operational event, not merely a code upload.

Provider traffic can point at the new Worker before every Sandbox reservation
observes the new container image. `--containers-rollout=immediate` does not make
those two completion proofs atomic. Release verification must keep the exact
Worker version pinned while it retries only explicit toolchain-convergence
codes. Forge bounds this to six probes with 110 seconds of total delay and
re-publishes `verifying` before each probe because a failed probe moves control
to `failed`. Any different Worker version, non-toolchain binding failure, or
exhausted budget fails the release and enters the normal rollback path.

The image cannot define which digest an arbitrary Worker version validates.
Current Workers publish `readiness.observed_toolchain_digest`; Deploy control
persists that exact value. Compatibility with an older Worker is a field-
absence fallback to its checked esbuild digest, including when that Worker sees
a v2 image. If a Worker publishes the field with an invalid value, verification
fails closed rather than silently selecting a digest from image metadata.

Required upgrade behavior:

- mark the runner as `draining` before the rollout;
- stop assigning new jobs until a health probe reports the new Worker,
  container image, fixed toolchain, R2 binding, and runtime-service binding;
- let in-flight jobs finish or checkpoint them;
- retry known transient reset/restart errors with exponential backoff and an
  idempotent run token;
- publish `ready` only after a real synthetic static build and dynamic bundle
  check pass;
- route failed post-upgrade jobs to retry rather than immediately labelling
  their user deployment terminally failed.

The hard-coded esbuild executable path should become a checked startup
assertion. Better still, the trusted toolchain should expose one Forge-owned
wrapper command whose version and digest are recorded; bundling code should
not rely on an undocumented global path.

### 5. Validate the effective build plan early

Deployment UI and API should show, for the exact commit:

| Field | Examples |
| --- | --- |
| Build command | `none`, `npm run build`, `node scripts/build.mjs` |
| Output root | `dist`, `build`, or manifest asset source |
| Manifest source | `forgeBuild.ts` path and normalized digest |
| Entrypoint | absent for static-only; source file for an application |
| Requested capabilities | assets, HTTP, durable state, realtime |

Before allocating a runner, Forge should reject obvious contradictions:

- a nonempty build command requiring a package manager but no matching
  project file at that SHA;
- a declared output root missing after an explicit no-build path;
- an application entrypoint absent or non-regular;
- manifest assets/fallback paths that are not present after build;
- configuration changes made after the deployment was queued.

This turns a remote runner failure into a clear configuration diagnosis.

### 6. Harden trusted ingest and artifact publication

The fixed-length upload requirement is correct: the trusted ingest path should
know the byte count before R2 accepts a stream. Keep these invariants:

- quiesce the build sandbox before trusted reads;
- verify each regular file’s observed size and digest after quiescence;
- use an explicit fixed-length stream (or an equivalent verified byte buffer)
  for every object upload;
- await both source-to-upload piping and R2 completion;
- verify returned object size and record ETag/digest in the manifest;
- write the completed artifact manifest only after every object is present;
- make retries reuse deterministic prefixes and safely overwrite or verify
  the same object identity.

Add a remote integration test that uploads a streamed artifact through the
actual Worker binding. Local mocks alone cannot detect provider stream
semantics.

### 7. Separate and reconcile lifecycle states

Forge should explicitly present these independent facts:

```mermaid
stateDiagram-v2
  [*] --> RefUpdated
  RefUpdated --> HandoffQueued
  HandoffQueued --> DeploymentCreated
  DeploymentCreated --> Building
  Building --> Ingesting
  Ingesting --> ProviderPublished
  ProviderPublished --> Verified
  Verified --> PreviewReady
  PreviewReady --> Active: auto publish or promote
  Building --> Failed
  Ingesting --> Failed
  ProviderPublished --> Failed
  Failed --> Retrying
  Retrying --> Building
```

For each edge, retain a timestamp, attempt number, idempotency key, and a
sanitized diagnostic. The UI must never collapse `RefUpdated` into
`PreviewReady` or `Active`.

## Observability and operator experience

### One deployment evidence record

For every deployment, generate a compact support bundle containing:

- repository, branch/ref, exact SHA, actor, and Git receive request ID;
- outbox event and handoff attempt history;
- deployment/workflow/job/step IDs and current states;
- configuration, manifest, plan, and policy digests;
- API, runner, edge, image, and provider version identities;
- artifact prefix, file count, bytes, and manifest digest;
- preview and production endpoint status checks;
- runtime script name/version when dynamic code exists;
- quota counters and suspension/access epochs;
- sanitized errors, retry classification, and next operator action.

Provider identifiers belong in this advanced evidence record; Forge IDs remain
the primary product handles.

### Metrics and alerts

Track at least:

- `git_ref_to_deployment_created_seconds` and the count of eligible refs with
  no deployment after the service-level objective;
- deployment success rate and terminal failure rate by phase;
- runner image/toolchain mismatch count;
- Durable Object reset/restart retry count;
- artifact ingest size/stream/manifest failures;
- provider publication and activation latency;
- live endpoint smoke success, separately for static, HTTP, state, and
  realtime fixtures;
- retry attempts, retry exhaustion, and reconciler repairs.

Alerts should distinguish a single repository configuration error from a
global API-binding, runner-image, R2, dispatch, or edge outage.

### Usage summary and traffic dataset

`GET /api/admin/deploy-reliability/usage` is restricted to platform Site
administrators. It reports the durable D1 control-plane totals for repository
event and deployment queues, successful deployment artifact files/bytes, and
deployment counts and reuse totals by `execution_path`.

Request and bandwidth usage has a different owning surface and is not copied
into D1. The Sites edge writes one point per response to the Cloudflare
Analytics Engine dataset `forge_sites_usage_preview` through the `SITE_USAGE`
binding:

| Slot | Meaning |
| --- | --- |
| index | Site ID, or `platform` before a Site is resolved |
| blob 1 | Deployment ID |
| blob 2 | `production`, `preview`, or `platform` |
| blob 3 | Request class: `asset`, `application`, `realtime`, `client`, `health`, or `platform_error` |
| blob 4 | Sites Worker version ID |
| double 1 | Request units, currently `1` |
| double 2 | Response bytes from trusted `Content-Length`, or `0` when unknown |
| double 3 | Edge handling duration in milliseconds |
| double 4 | HTTP response status |
| double 5 | Dynamic request units, `1` for application/realtime and otherwise `0` |

Cost and bandwidth reporting must query this dataset by Site and time window.
The D1 endpoint intentionally returns the dataset contract rather than
presenting incomplete request or byte totals as authoritative.

## Security and abuse boundaries

Reliability machinery can itself become an abuse multiplier, so the trial
slice applies the following limits:

- Ref triggers create outbox work only for enabled Sites. Ordinary repository
  pushes do not consume Deploy outbox storage.
- Dispatch and reconciliation batches are capped at 25 records per pass.
  Leases expire after 60 seconds; retries stop after eight attempts and use
  exponential backoff capped at 15 minutes.
- Dead-letter events are retained for diagnosis rather than retried forever.
  The operator view reports backlog and incident counts.
- Git-receive annotations consider at most 100 branch changes per request.
- Operational messages are length-bounded and redact bearer tokens, common
  secret assignments, and URL credentials before persistence or logging.
- Runner state and release controls require an authenticated Site operator.
  A direct transition to `ready` is forbidden; readiness requires the deep
  runner probe.
- Deployment evidence is visible only to repository administrators because it
  contains internal version and provider identifiers. Public hosted output
  does not expose it.
- Health endpoints disclose contract/version state but never secrets. Missing
  required bindings fail closed.

Before broader self-service, add retention for delivered/dead-letter outbox rows,
per-repository handoff rate metrics, anomaly alerts for repeated ref churn,
and a global emergency pause that halts dispatcher claims without deleting
intent. These are capacity and abuse controls, not substitutes for existing
repository authorization or Sites suspension.

## Staged verification plan

Run these in order so a failure is attributable and rollback remains small:

1. **Database contract:** apply all five canonical fresh-schema fragments to an
   empty local D1 database, verify foreign keys/triggers/default runner state,
   then update one enabled project ref and confirm exactly one pending event is
   produced. No upgrade or ledger-adoption path exists.
2. **Control-plane contract:** deploy the API with its full binding manifest,
   inspect `/health`, run one normal push, and verify the ref event, deployment,
   workflow, and handoff timeline share the expected SHA.
3. **Recovery contract:** temporarily block runner admission, push twice, and
   prove intents remain pending with bounded retries. Restore readiness and
   prove each intended SHA creates at most one deployment.
4. **Release contract:** upload—but do not activate—a runner version/image,
   set `draining`, wait for active work to settle, set `verifying`, and invoke
   the deep probe. Only a matching Worker/image/toolchain tuple may become
   `ready`.
5. **Execution-evidence contract:** deploy one fixture and compare the evidence
   panel with Cloudflare Worker versions, the runner image digest, dispatch
   namespace, deployment record, and live hostname.
6. **Chaos contract:** repeat with a Git disconnect, an expired dispatcher
   lease, a runner reset during setup, a reset after repository code starts,
   an R2 interruption, and a missing API binding. Confirm safe replay where
   possible and an explicit blocked/terminal state everywhere else.
7. **Abuse contract:** generate rapid ref churn for one allowlisted fixture,
   verify batch/retry caps and no duplicate deployments, then suspend the Site
   and confirm new intent is not created or dispatched.
8. **Conformance contract:** rerun static, reservations, and realtime fixtures;
   verify SPA fallback, exactly-one reservation claims, state persistence
   across code rollback, realtime resync/quota refusal, egress denial, and
   immediate disablement.

### Fresh D1 schema

Forge has no production data contract yet. Schema changes replace the five
size-bounded fragments of one fresh-install schema and require recreation of
the pre-production D1 database. The verifier applies those fragments to an empty database and fails on
integrity, foreign-key, required-object, seed-signature, or forbidden
compatibility-object mismatches. No ledger-adoption or upgrade procedure exists.

## Acceptance and chaos checks

Before opening the trial beyond the allowlist, automate these remote checks:

1. Push each fixture and prove a deployment record appears within the handoff
   SLO, even if the Git client disconnects immediately after refs update.
2. Deploy an API version missing a required hosting binding in a staging
   environment; prove an eligible Site creates an observable configuration
   incident rather than silently doing nothing.
3. Roll the runner Worker and container image during queued fixture builds;
   prove jobs drain or retry and no run is abandoned.
4. Verify a runner startup probe finds the pinned esbuild wrapper and performs
   a minimal bundle before dynamic jobs become eligible.
5. Exercise streamed asset uploads against real R2 with zero-byte, small,
   multi-megabyte, and interrupted files; verify retry idempotence.
6. Configure an assets-only fixture with no build command; prove it deploys.
   Configure a stale command; prove Forge rejects it before runner allocation.
7. Confirm static SPA navigation uses an HTML navigation request and falls
   back to `index.html`, while an asset fetch still returns a real 404.
8. For reservations, send concurrent claims and prove exactly one succeeds;
   roll back code and prove production durable state persists.
9. For realtime, open multiple tabs, verify update/resync behavior, quota
   refusal, suspension, and reconnect after a runtime rollout.
10. Disable a Site and prove edge serving, runtime calls, tickets, mutations,
    and sockets stop immediately.

## Prioritized backlog

### P0 — before more trial users

- [x] Database outbox and reconciler for ref-to-deployment handoff.
- [x] Required-binding release manifest and deployment-time verification
  (local; staged activation rehearsal pending).
- [x] Fail-loud trigger behavior for eligible Sites with missing hosting
  configuration.
- [x] Runner rollout drain/health gate and retry classification for code-reset
  errors.
- [x] Record API, runner, image, and edge versions on every new deployment.
- [x] One repository-admin deployment evidence view in the UI.
- [ ] Persist live-hostname smoke checks in the deployment evidence record.

### P1 — before self-service expansion

- [ ] Effective-build-plan preview and preflight validation.
- [x] Pinned, self-verifying Forge toolchain wrapper; remove reliance on an
  opaque global executable path.
- [x] Remote integration exercise covering R2 cleanup, dynamic dispatch,
  runtime state, realtime, disablement, and rollback.
- [ ] Phase-specific retry policy, retry budget, and actionable surfaced
  errors.
- [ ] Deployment SLOs, dashboards, and global-versus-project alert routing.

### P2 — production graduation

- [ ] Separate registrable hosting domain, TLS/DNS rollout and rollback
  rehearsal.
- [ ] Multi-provider adapter health/evidence model before Railway or Modal
  workloads are enabled.
- [x] Automated preview retention and artifact garbage collection.
- [ ] Full cost attribution and abuse operations.
- [ ] Full disaster-recovery exercise for control-plane records, immutable
  artifacts, and runtime state boundaries.

## Session takeaways

1. The overall architecture is viable, but availability comes from the
   handoffs between Forge services, not from any single Worker succeeding.
2. Exact-SHA execution is necessary but insufficient: the exact control-plane
   configuration, code version, image, toolchain, and provider state must be
   attributable too.
3. “Retry” is a valid recovery tool only when it is deliberate, idempotent,
   classified, and visible. Manual empty commits are acceptable during a
   tightly controlled trial, not as a product mechanism.
4. The three fixtures are valuable as permanent conformance probes. Keep them
   deployed and run them after every control-plane, runner, edge, or provider
   rollout.