SmolForgeField Notes

Notes on systems,
code, and craft.

All notes

Engineering

Projected Checkpoints Cut Checkout 96%

Declared build context cut source bytes 87%; direct-parent deltas cut traversal 98%.

Forge reduced OverGrid’s exact-SHA checkout from 366.931 seconds to 14.701 seconds by restoring a projected parent and applying only changed Git objects.

Halfway through the investigation, the honest status was “not done yet.” The first cold checkpoint seed had hit the 300-second deadline, the Git server was still spending most of its time constructing a roughly 204 MB pack, and the hoped-for fast child run could not start until one parent seed survived.

A coding-agent conversation in dark mode. The user asks how it is going; the response says the work is not done, identifies a 300-second cold-clone deadline and a slow 204 MB Git pack, and says production speedup proof is still pending.
Thread checkpointLatest visible state: not done yet. […] The root issue is confirmed: Forge’s Git server spends too long constructing OverGrid’s ~204 MB pack with no output.Authentic Codex side conversation captured during the production canary; excerpt reproduced semantically for legibility.

The finished path cut OverGrid's connected Deploy source checkout from 366.931 seconds to 14.701 seconds, a 96.0% reduction, without weakening the exact-commit guarantee. Forge reduced the declared source boundary, restored the matching projected parent, and applied only the changed Git objects.

The decisive measurement was not the final duration. A projected child initially spent 26.438 seconds walking the complete selected tree but only 1.464 seconds materializing it. Parent reuse cut that traversal to 452 milliseconds, or 98.3%, while the source archive shrank from roughly 180.25 MB to 22.69 MB. A concurrent generic Action still needed the complete repository and spent 211.627 seconds in checkout; these are production canaries for two different input contracts, not a general latency distribution.

Exact-SHA caching was correct but incomplete

Forge's first exact-source cache already keyed artifacts by repository, exact 40-character commit SHA, schema, project root, and archive format. A hit could not drift with a branch. A miss fell back to exact Git transport.

That contract made repeated builds predictable, but every normal push creates a new SHA. OverGrid's repository was about 204 MB. Its historical cold seed took 196.829 seconds, including 178.500 seconds for the clone and 15.564 seconds to publish the checkpoint.

The first attempt restored the direct parent's checkpoint, then asked Git Smart HTTP for the child commit. For a one-file change, the server still walked and packed 1,561 objects. That “delta” fetch took 118.229 seconds, and checkout still took 168.700 seconds.

The cache identity was sound. The work performed after a miss was not proportional to the change.

Put source selection in the build contract

OverGrid contains generated docs/ai/** research that the application build does not need. Eleven JSON receipts inside that directory are imported by application code. The project now declares that boundary beside its build command in forgeBuild.ts:

build: {
  command: 'pnpm run build',
  context: {
    exclude: ['docs/ai/**'],
    include: [
      'docs/ai/artifacts/ai-program-summary.json',
      // Ten more application inputs.
    ],
  },
}

The policy starts with the complete project, applies exclusions, then restores explicit exceptions. Forge also forces the build configuration, referenced Wrangler file, package manifests, lockfiles, workspace dependencies, and project root into the selected closure.

Forge reads this policy from the requested commit before checkout. Its normalized digest becomes part of the immutable v4 cache key. Changing the policy therefore cannot reuse a checkpoint created for a different source set. Omitting the policy still means complete project source.

This is a declared-input contract, not import inference. Docker removes ignored files from a selected build context, while Bazel keys remote-cache entries from declared action inputs. Forge uses the same safety principle without asking projects to adopt another build language.

Reuse one projected parent

Projection alone reduced the payload to 1,539 Git objects and 22,687,273 compressed bytes, but the first projected children still took 40.227 and 48.457 seconds. Phase-level profiling explained why: Forge rebuilt the projected closure for every new commit.

The corrected path reuses a checkpoint only when the target has exactly one parent and that parent's checkpoint has the same projection digest. Forge then walks changed paths between the two trees and creates a bounded layer containing:

  • the target commit;
  • changed ancestor trees;
  • changed or newly reachable blobs; and
  • Git's native compressed object bytes.

The checkout path stays explicit:

requested exact SHA
  ├─ target checkpoint exists → restore target → verify
  └─ target checkpoint missing
       ├─ matching direct-parent checkpoint exists
       │    └─ restore parent → apply changed objects → verify
       └─ no eligible parent → bounded cold exact checkout → verify

The sandbox validates archive members and declared bounds, imports the objects, rebuilds only the projected index, and verifies the target SHA and selected tree before repository code runs. Tests cover file additions and deletions, executable-mode changes, symlinks, and file-to-directory transitions. A changed projection, merge commit, invalid checkpoint identity, integrity error, or exceeded bound cannot enter the fast path. Structural ambiguity in an excluded path falls back to a fresh projected traversal rather than guessing.

Each successful target becomes another complete immutable checkpoint. Forge never follows a recursive chain of deltas, so deleting an older parent cannot strand a newer target. The checkpoint remains derived acceleration state; repository objects remain the authority.

Profiling rejected the tempting fix

The first fresh projected-child receipt took 38.763 seconds. Tree traversal consumed 26.438 seconds, archive transfer 9.244 seconds, and materialization 1.464 seconds. That split ruled out a tempting shortcut.

Cloudflare Sandbox backups restore through a SquashFS/FUSE overlay. A snapshot could help when extraction dominates, but it would have attacked roughly one or two seconds here while leaving the 26-second walk intact. The measured bottleneck was repeated object discovery, so Forge reused the projected parent's object closure instead.

On the final one-line canary, changed-path traversal fell to 452 milliseconds. Restoring the parent took 6.717 seconds, the four-object child archive transferred in 67 milliseconds, and parent-plus-delta materialization took 2.880 seconds. The runner's inner source phase reported 14.357 seconds; the enclosing exact-SHA checkout receipt reported 14.701 seconds.

One common scale: checkout fell 96.0%, source bytes fell 87.4%, and direct-child traversal fell 98.3%. Immutable parent transfer is now larger than discovery or delta application.

The figure is a comparison between one original full-source receipt and one direct-child production canary. It does not establish a percentile or promise the same duration for repositories with different trees, contexts, or cache state.

Freeze once, publish while the build runs

The first fast child exposed another trap. Checkout succeeded, but the optional publisher performed a second full traversal of the target and stalled. A fast consumer had recreated the expensive producer work.

Forge now freezes the already verified sandbox object database before repository code runs. It removes duplicate downloaded tar files, deterministically archives the unique parent-plus-delta objects, and hashes fixed 16 MiB parts. Publication uses digest-fenced multipart upload through the Worker's R2 binding and overlaps install, build, and tests. A changed part cannot be published under the prepared checkpoint identity.

Run #307 published before the application build ended (source_seed_join_ms=0). Run #308 then restored that exact target with zero traversal. The repeat took 18.188 seconds in the runner source phase; transferring the immutable target, not rediscovering its graph, dominated the result.

Push fan-out also needs restraint. A Sites consumer waits for an active sibling Action only when the target is a default-branch push, a complete compatible parent exists, and that sibling is producing the same repository and SHA. The wait is bounded at 90 seconds. A truly cold repository skips it, so coordination cannot add a wait before the same cold checkout.

The fast path has a deliberate edge

forgeBuild.ts declares build inputs. It does not describe arbitrary shell commands. OverGrid's generic Action runs 321 test files; 35 refer to generated docs/ai evidence beyond the 11 receipts required by the application build. Reusing the Deploy projection would make that workflow faster by making the test run incomplete.

The safe extension is an explicit run-level source contract, defaulting to full source. A lean deploy-validation Action could opt into the build context, while scheduled research-integrity checks keep the complete repository.

Full-source work also has a different remaining bottleneck. The same changed-path algorithm cut one Action's tree walk to 481 milliseconds, but moving its 180.25 MB parent still took 127.311 seconds. Total checkout was 145.612 seconds, only 18.6% better than its prior same-SHA baseline. Faster transport or a native snapshot would address measured work there; another traversal optimization would not.

This change also does not accelerate external git fetch, and it does not make OverGrid's 1.03 GB pnpm store worth restoring. Forge keeps that dependency cache above its 512 MiB ceiling and runs the smaller frozen install instead.

Cache the contract, not the branch

The durable rule is compact:

A build receives the exact requested commit and its declared source context. The normalized context is part of immutable cache identity; no explicit context means the complete project.

That invariant let Forge remove 96% of checkout time without weakening source correctness. The next test is no longer whether smaller Git deltas work. It is whether full-source archive transport and explicit Action contexts can improve the remaining slow paths without turning an optimization into hidden authority.

The public release boundary remains the Forge Deploy specification, with agent-readable guidance in llms.txt.