Forge container CI/CD#

Forge can execute runner: container workflows through a private Cloudflare service binding. A successful push records the exact commit SHA, starts a durable Cloudflare Workflow, checks out that SHA in a fresh Sandbox container, and executes run: steps in order.

Fresh public-repository checkouts first look for a private immutable source archive keyed by internal repository ID and exact commit SHA. A valid archive downloads onto native ephemeral storage and is checked with Git before use. On a miss, Forge elects the earliest Action for that push as the source producer. Later Actions and a root Sites build poll for its immutable checkpoint for at most 90 seconds, then fall back independently if the producer stalls or fails. The producer performs one bounded exact-SHA Git checkout, repacks that verified checkout into a deterministic loose/pack-object v2 archive, and uploads it directly from the sandbox. This target wait does not require an older parent checkpoint. Manual, pull-request, and other read-only runs never mutate the shared source cache. The complete commit closure never silently omits docs/, generated assets, or any other tracked directory.

Private connected builds follow the same cache-as-acceleration rule. Forge first checks for an immutable repository+SHA+project-root archive. A cache hit uses a signed bulk download into the credential-free sandbox. On a cold miss, Forge authorizes and verifies the exact project closure, then streams that archive directly into the sandbox; it does not block the build on uploading a second derived cache copy. A later trusted seed may populate the immutable archive, while unchanged successful build artifacts remain reusable by their full artifact identity.

If the exact tree exceeds the direct materializer's documented object, byte, or depth limits, Forge falls back to a shallow fetch of the recorded exact commit. Actions recreate their local branch and tracking refs around that immutable SHA instead of cloning whatever the mutable branch points to later. Every network exact-SHA Git checkout attempt is hard-limited by CI_CHECKOUT_ATTEMPT_TIMEOUT_SECONDS (600 seconds in production, 120 seconds when unset, and clamped to 30–600 seconds). Exact-source archive lookup, trusted seed publication, transfer, and materialization use the separate CI_EXACT_SOURCE_ATTEMPT_TIMEOUT_SECONDS ceiling (1,200 seconds in production, clamped to 30–1,800 seconds and falling back to the checkout setting). Cold preparation emits bounded object-discovery, transfer, sandbox-validation, and checkout-materialization stages; repeat SHAs use a verified build artifact or an existing immutable source archive. This keeps derived-cache writes and warm validation from expanding the independent Git transport budget. A hung path is terminated and recorded as an explicit optimization failure instead of leaving the run silently stuck. The durable Workflow owns infrastructure retries after both source paths fail; the shell checkout deliberately does not multiply that budget with another retry loop. The same bound covers legacy restored-workspace reset and cleanup, so a damaged or oversized cache cannot strand the runner before a clean clone begins. Cleanup empties the restored directory without deleting its Sandbox mount point. The watchdog also gives a newly errored Workflow two minutes to consume its configured infrastructure retry budget before reconciling the run to failure.

Large exact trees can require thousands of R2 reads plus CPU-heavy object inflation and digest verification. The production runner explicitly allows 300 seconds of CPU and 10,000 subrequests in packages/runner/wrangler.toml; removing those limits can exhaust Workflow defaults before dependency detection even when the repository is within every materializer bound.

The direct materializer's current safety bounds are:

These are runner memory and execution guards, not source-retention rules. An eligible fallback checkout still materializes the complete tracked tree.

Sites projects may additionally declare build.context in forgeBuild.ts. Forge statically reads that policy from the authorized exact commit before the first source transfer, includes its canonical digest in the v4 checkpoint identity, and materializes only the selected project paths plus required build controls and workspace dependencies. A custom context deliberately does not reuse the direct-parent delta path in its first release; cache misses traverse the projected exact tree and publish a projection-scoped checkpoint.

Forge advertises reachable and arbitrary object wants with partial-clone filtering, matching its repository-scoped object server behavior. Git can then materialize promised blobs from the selected commit in a bounded follow-up pack instead of rejecting each blob as unadvertised.

Multi-round promised-object negotiation terminates every ACK/NAK response with a packet flush, allowing large sets of missing blobs to span requests without being misread as a truncated pack stream.

Promised-object requests without have negotiation generate their pack immediately even when Git omits an explicit done packet, as required by the stateless Smart HTTP exchange used during sparse checkout.

Git gzip-compresses larger Smart HTTP request bodies, including batched promised-blob wants. Forge decodes that transport encoding before pkt-line parsing, rejects corrupt gzip with HTTP 400, and rejects unsupported encodings with HTTP 415 instead of silently treating the request as an empty negotiation.

Large promised-object packs are generated and checksummed incrementally, with only one object retained at a time. Stateless filtered wants skip a redundant object-graph traversal. This keeps full working-tree materialization below the Worker memory ceiling even for repositories containing large tracked artifacts.

Git pull telemetry records only aggregate request shape (byte size and counts of wants/haves, encoding, decoded size, and negotiation flags), never object IDs or credentials, so protocol failures can be diagnosed without leaking repository contents. Pack-generation failures emit a structured error event and fail the Git stream instead of remaining invisible behind a malformed response.

JavaScript package-manager recommendation#

Forge strongly recommends pnpm as the default for new JavaScript and TypeScript projects. Bun is the high-speed option when project compatibility allows. Existing npm and Yarn repositories remain supported and should not be silently migrated by a workflow.

Commit a packageManager field plus exactly one matching lockfile. New pnpm workflows should use a frozen-lockfile install such as pnpm install --frozen-lockfile; Bun projects should use bun install --frozen-lockfile or bun ci. The standard container image includes pinned pnpm and Bun toolchains, so projects do not need to download a package manager before every job.

Security boundary#

• Container workflows currently accept public repositories only.
• The checkout is verified against the SHA received by Git Smart HTTP.
• A deploy step is rejected when its branch has advanced to a newer SHA.
• The account API token never enters the repository-controlled container.
• Cloudflare API egress is limited to the configured account and Worker script.
• uses: steps, mixed runner types, and private repository checkout fail closed.
• Step logs are capped at 2,000 lines and 256 KiB.

Cross-build caching#

Container jobs default to cache: auto. Forge detects the declared packageManager field or the sole matching lockfile and restores only that manager's download/store cache: pnpm's store, Bun's install cache, or npm's content-addressed cache plus content-keyed tool caches such as ESLint. An ambiguous or unsupported manager (including Yarn) is skipped safely rather than receiving an npm cache by accident. Keeping cache: auto in authored YAML is still useful when the workflow should state the policy explicitly.

cache: npm remains the explicit npm download/tool-cache mode. cache: npm-exact is the legacy, npm-only opt-in which also restores node_modules when its full dependency fingerprint matches. Generated Sites builds use the same internal exact-snapshot representation for npm, pnpm, or Bun when cache: auto resolves one authoritative manager and lockfile; repositories do not add another configuration field.

Exact dependency snapshots are capped at 512 MiB of compressed payload by default (CI_EXACT_CACHE_MAX_BYTES). Forge captures a versioned SquashFS image immediately after its frozen install and before repository build code runs. This preserves every root and workspace-local node_modules tree—including executable modes, symlinks, hardlinks, type declarations, and native packages—without capturing repository source or letting a build mutate the reusable snapshot. The key includes repository, exact source SHA, normalized project root, selected manager and lockfile, runtime, platform, architecture, libc, install policy, runner image, and snapshot layout. The package-manager cache remains the fallback if compression or restore fails.

One push also has one preferred exact dependency-cache producer. A connected Site publishes first because provider ingest already depends on its frozen installed tree; otherwise the earliest Action publishes. Sibling Actions poll for that immutable row for at most 60 seconds before falling back. The lookup happens before SquashFS preparation, so a sibling records action=prepare outcome=skipped reason=already_published instead of spending another minute recompressing hundreds of megabytes. The save path rechecks the immutable fingerprint after preparation, preserving safe convergence if a producer finishes during the fallback.

When the compressed snapshot exceeds the cap, Forge logs size_limit_exceeded, keeps the normal manager cache, and runs the workflow's usual install command. Existing exact snapshots remain reusable until expiry; the cap only prevents new oversized uploads. Raise the variable only when the measured restore-plus-skipped-install benefit justifies the storage and upload cost. Do not raise CI_MANAGER_CACHE_MAX_BYTES as a substitute: that restores the large store through the lazy FUSE path that this circuit breaker avoids. Use CI_EXACT_CACHE_PROJECT_LIMITS for measured exceptions; it is an operator-owned JSON object from Forge project ID to compressed bytes, for example {"dprj_example":805306368}. Invalid entries fail back to the global 512 MiB ceiling, and both save and restore enforce the project limit.

jobs:
  build:
    runner: container
    cache: auto
    steps:
      - name: Install
        run: pnpm install --frozen-lockfile
      - name: Build
        run: npm run build
      - name: Test
        run: npm run test:unit

Each manager-cache key includes that manager's own lockfile, chosen manager and version, package metadata, .npmrc, Node runtime, platform, architecture, libc, frozen-install policy, and runner image. Cache restore and save errors are soft failures so the workflow can still perform a clean install.

Manager stores are repository-scoped, untrusted content caches rather than Forge-authored dependency artifacts. Before repository code runs, Forge uses the npm or pnpm manager's own integrity check; an invalid store is cleared without removing its Sandbox mount point. Bun has no equivalent preflight, so its normal frozen install remains the verification boundary. Forge never replays a failed authored install automatically because lifecycle scripts may already have produced side effects. If Forge cannot establish a clean manager directory before any authored step starts, setup fails and the durable infrastructure retry uses a fresh Sandbox.

Production Sandbox backups restore as SquashFS/FUSE copy-on-write mounts. Forge therefore distinguishes a mounted cache from an extracted cache in runner logs, and caps manager caches at 512 MiB of logical content by default (CI_MANAGER_CACHE_MAX_BYTES). A larger or unmeasurable manager cache is reported as skipped reason=bypassed_by_size (or size_measurement_failed) before mount/save. This protects execution time from large lazy-hydration regressions while preserving the frozen clean install.

Forge exposes cache state to project commands:

• FORGE_EXACT_CACHE_HIT=1 means the matching extracted node_modules tree is already present. Generated Sites builds skip their Forge-owned frozen install; authored cache: npm-exact workflows must continue to skip npm ci, which would otherwise delete it.
• FORGE_DEPENDENCY_CACHE_HIT=1 means the selected pnpm, Bun, or npm dependency backup mounted or extracted successfully; installs must still use their normal frozen/clean behavior, and end-to-end timing determines whether the cache was useful.
• FORGE_NPM_CACHE_HIT=1 remains available for explicit npm cache workflows.
• NPM_CONFIG_CACHE, PNPM_STORE_DIR, BUN_INSTALL_CACHE_DIR, and ESLINT_CACHE_LOCATION point at Forge-managed cache directories. Project scripts should honor these paths instead of hard-coding repository-local cache locations.

For package scripts, use the same contract in a reusable command such as ci:install, and keep unit tests separate from build so the workflow builds once rather than rebuilding implicitly.

For generated OpenNext Sites, Full rebuild current SHA is the explicit same-commit dependency-warm benchmark and recovery path. A project administrator request bypasses the finished provider-artifact shortcut, may restore the exact source and exact dependency snapshot for that unchanged SHA, and still executes the complete OpenNext build. Runner logs record prebuild_reuse outcome=skipped reason=manual_full_build. This does not restore Next.js or OpenNext build output, and it is not evidence of an ordinary new-commit warm build.

Exact snapshots deliberately include the entire source SHA in their identity. Package lifecycle scripts execute during the frozen install and can read or derive installed bytes from arbitrary repository source, including monorepo siblings. Reusing that post-install tree for a different SHA would weaken build correctness. Ordinary new-SHA pushes can reuse only the selected manager store and exact-source projections; a future cross-SHA install cache needs an isolated, reproducible lifecycle-script boundary rather than a broader key.

Only trusted default-branch identities can write caches: a push, or an authorized Site manual/redeploy request pinned to the current production branch SHA. Pull requests and historical preview rebuilds remain read-only. Every cache transfer temporarily allows only the R2 storage hostname and removes that egress rule afterward, for Actions and Sites alike. Transient provider failures use bounded jittered retries. Before publishing an immutable dependency or manager cache, Forge rechecks its content-addressed record and skips the upload when another run has already completed it. When exact Git objects are available, Actions bypass prepared-workspace snapshots on the direct path. If the exact commit exceeds a materializer bound, an Action may restore its prepared workspace before using network checkout. environment files, logs, build output, dependencies, and tool caches are excluded from those snapshots.

Each run records cache kind, hit/miss/skipped/error outcome, duration, bytes, and fallback reason in ci_cache_events. Structured [runner:cache] logs also distinguish restore_method=fuse_overlay from restore_method=extracted. Checkout logs separate source-archive lookup, authoritative object traversal, archive transfer, materialization, and exact-HEAD verification. Post-terminal logs report cache-save time and total runner occupancy separately from user-visible workflow execution.

The Actions run page classifies dependency work as cold, mounted, warm, partially warm, bypassed by policy, or not configured and compares execution time with up to five recent successful runs of the same workflow and branch. Mounted means the backup became accessible; it does not claim the subsequent install was faster. Agents can retrieve the same evidence and recommendations with:

smolforge actions inspect owner/repository RUN_ID
smolforge actions inspect owner/repository RUN_ID --json

The default output is concise human-readable evidence; --json preserves the complete structured contract for automation.

Deployment#

1. Apply D1 migrations with the API Wrangler config only when the release actually includes schema changes.
2. Set the runner Worker secret CI_CLOUDFLARE_API_TOKEN to a scoped API token.
3. For cross-build caching, create an R2 bucket, bind it as BACKUP_BUCKET, and set BACKUP_BUCKET_NAME, R2_ACCESS_KEY_ID, and R2_SECRET_ACCESS_KEY. Scope the S3-compatible token to object read/write on that bucket only and configure object expiration slightly beyond CI_CACHE_TTL_SECONDS.
4. Require a clean, pushed commit, ready runner control, and exactly one matching provider version at 100%. Set control to draining; the release coordinator polls the authenticated drain endpoint and refuses provider upload until queued and in-progress container work reaches zero. The wait is bounded to 31 minutes and fails closed. Do not delete stale queued records as part of a code release. Draining closes only long-running job admission. The exact, already-verified runner continues accepting idempotent staged Git ingest until provider traffic moves away from it.
5. Upload the runner with wrangler versions upload --strict --message <sha> --tag forge-runner-<short-sha>, then resolve and inspect that exact version's annotations, bindings, and variables. Container-backed uploads may activate immediately, so read provider traffic: accept the exact new version at 100%, or explicitly deploy it only if the old version remains at 100%. Reject any foreign, mixed, or malformed traffic state.
6. Require the inspected version at 100% before continuing. Stateful Workflow/Durable Object runners do not use a traffic split. Set control to verifying, pin the Worker/image/toolchain tuple, run the deep service-binding probe, and require the matching tuple to return ready. The checked toolchain contract includes esbuild 0.25.8, Wrangler 4.113.0, SQLite 3.37.2, their exact binary digests, and an aggregate digest. Use scripts/forge-release-component.mjs --component runner --sha <sha> with SMOLFORGE_ADMIN_SESSION set; the command drains admissions before upload, verifies the exact Worker/image/toolchain tuple, and restores the rollback target if activation or verification fails. Worker traffic activation and container-image convergence are separate provider events: a newly active Worker can briefly observe the prior image. The gate therefore retries only toolchain failures reported by the exact expected Worker version, using six probes over a bounded 110-second delay budget. Every retry re-enters verifying; version drift, authority errors, and unrelated binding failures fail immediately. While the successor is verifying, the repository Worker retains staged Git packs and retries transient private-runner handoff failures; refs move only after the exact runner returns a durable receipt. The Worker reports observed_toolchain_digest, the exact digest its own readiness code enforced, and Deploy control persists that value. For an old Worker without this field, Deploy control falls back only to the esbuild digest that old Worker code validated—even if the converged image already emits a v2 contract. A malformed reported digest fails closed instead of downgrading to the compatibility path. A full image rollout first performs a read-only registry listing to prove containers:write; missing authority fails before the runner is drained or a Worker is uploaded.
7. Build fresh web assets and deploy @cloudforge/api from the same source SHA. Re-open the active Cloudflare versions and public health endpoints.
8. Record runner, API, Sites edge, deployment runtime, Wiki, and AI router releases as active, then require exactly one matching active ledger row for each deployed service. Activation atomically retires prior active rows for that service/environment while preserving verification and activation timestamps.
9. Run a cold canary and at least two same-SHA warm canaries. Inspect the owning Action records through post-run cache save and Sandbox destruction, and verify the deployed application's live health after each run.

The runner has workers_dev = false; it is reachable only through bindings. Repository-to-script deployment grants live in CI_DEPLOY_POLICIES in the runner Wrangler configuration. Adding another repository requires an explicit policy entry and corresponding egress validation.

The public Forge /health endpoint checks the private runner binding and returns HTTP 503 with status: degraded when the runner is missing or unreachable. Health responses are never cached, keeping CI control-plane outages visible to external monitors instead of replaying a stale green check.

External failure alerts#

Set the runner Worker secret CI_ALERT_WEBHOOK_URL to an HTTPS incident endpoint (for example, a dedicated Slack or PagerDuty webhook). When the runner records a terminal container-run failure, it sends one metadata-only JSON event:

{
  "event": "ci_runner_workflow_failed",
  "service": "cloudforge-ci-runner",
  "run_id": 91,
  "observed_at": "2026-07-23T00:00:00.000Z"
}

It deliberately excludes repository names, branch names, commit SHAs, workflow commands, logs, and failure text. Set the optional CI_ALERT_WEBHOOK_SECRET to add an x-cloudforge-signature HMAC-SHA256 header. Alert transport failures are logged without printing the endpoint or any workflow detail; Forge still records the terminal run failure.

npx wrangler secret put CI_ALERT_WEBHOOK_URL --config packages/runner/wrangler.toml
npx wrangler secret put CI_ALERT_WEBHOOK_SECRET --config packages/runner/wrangler.toml

The second command is optional. Do not put either value in wrangler.toml.

Failure and recovery#

A dispatch, checkout, command, stale-SHA, or policy failure is written to the existing workflow run/job/step tables and exposed by the Actions API. Sandbox containers are destroyed after each run. Reruns retain the original trigger SHA so they cannot silently build a different commit.

Transient infrastructure failures during sandbox setup or checkout are retried twice with exponential backoff. Forge does not automatically retry after a repository command starts, because an interrupted deploy or other external side effect may have completed even when its result was lost. That ambiguity is reported as a terminal failure instead of risking a duplicate mutation.

Git Smart HTTP reads and writes use separate rate-limit buckets. Public clone traffic has a high read budget so CI containers sharing Cloudflare egress do not exhaust the tighter mutation budget used by pushes.

The runner watchdog reconciles active Forge runs against Cloudflare Workflow state every minute. Errored, terminated, missing, or inconsistently completed instances are marked failed with their control-plane error. The Actions detail page polls active runs, displays the failure summary and last recorded runner phase, and reruns navigate to the newly created run.

Run success becomes terminal as soon as every declared job step succeeds. Dependency-cache uploads happen afterward as best-effort acceleration, and a late sandbox reset cannot overwrite a terminal result. If the Workflow runtime retries after all recorded steps succeeded, it finalizes the run as successful; mixed, failed, or partially executed step states still fail closed.

Repository command logs are bounded, but truncation preserves both the initial setup context and the final output where compilers, test runners, and deployment tools normally print their actionable error. The inserted marker reports the omitted line and byte counts instead of silently dropping the failure tail.

The runner's interruption contract is exercised as a chaos matrix: setup resets may retry, resets after repository commands begin fail closed without replaying side effects, and resets after every declared step succeeded preserve success.