Engineering
130 MiB Pushes Stay Ordinary
Stage the pack, validate objects, then compare-and-swap the ref.
Forge moved large-pack work out of the edge request, validated submitted packs with native Git, published immutable objects, and made each ref update a final compare-and-swap.
The constraint arrived as a product request, not a bespoke importer: stock Git clients had to feel responsive during clone, fetch, and push. A migration-sized repository still needed to use the same remote and the same git push command as a one-line change.
Forge now accepts a large repository through that ordinary protocol path. The important change was not a larger request limit: reception, validation, object publication, and ref movement became separate bounded steps.
One production acceptance run pushed 130.04 MiB in 85.73 seconds, then cloned the resulting ref in 22.65 seconds with an identical SHA-256 tree digest. That is one observation, not a latency promise. The durable result is the ordering rule behind it: a branch cannot move until the submitted pack has passed native Git validation and its objects have been published.
The edge request was the wrong owner
Forge exposes Git over HTTP from a Cloudflare Worker. The Worker is the right place to authenticate a caller, resolve repository policy, parse the requested ref changes, and speak Git's receive-pack protocol. It is the wrong place to accumulate and inflate an entire migration-sized pack.
The first receive path combined too much work in one request boundary. It buffered the incoming body, parsed Git objects in JavaScript, resolved deltas, wrote objects, and only then updated refs. Small pushes worked, but repository size translated directly into request memory and time. A failure could also arrive after the client had spent minutes uploading.
Increasing a body limit would have preserved that coupling. A special import API would have avoided the request, but it would also have created a second repository-ingest product with different authentication, progress, and failure behavior. We wanted one protocol path instead:
git remote set-url origin https://forge.smol.ai/<owner>/<repo>.git
git push -u origin main
The design question became: how little of a push must the edge request own?
The pack crosses a durable boundary
The API reads the bounded receive-pack command prefix first. That prefix says which refs the client wants to create, update, or delete and includes the expected old commit for each change. The remaining body is the pack stream.
Forge stages that stream as a repository-scoped multipart object. Parts are 16 MiB, with at most four uploads in flight. The code counts every byte while reading and rejects a pack above 490,000,000 bytes. The decimal limit is deliberate: it stays below the provider's 500,000,000-byte request ceiling, whereas “500 MiB” would exceed it.
While the trusted runner processes the staged pack, receive-pack sends an indexing heartbeat about every ten seconds. This is protocol progress, not a polling UI. A normal Git client can show that the server is still working without Forge inventing a separate import session.
The request now follows this sequence:
Git client
│
├─ command prefix ──> authenticate and authorize requested refs
│
└─ pack stream ─────> bounded multipart staging
│
▼
native Git validation
│
▼
publish immutable objects
│
▼
compare-and-swap each ref
The staging object is temporary. Forge deletes it after the runner returns, including on failure. An interrupted multipart upload is aborted. Retry therefore means the Git client submits the push again; it does not mean an abandoned stage can later publish a ref.
Native Git decides whether the pack is valid
The runner creates an isolated bare repository and executes the validation path with native Git:
git index-pack --fsck-objects --threads=2 --max-input-size=490000000 incoming.pack
git unpack-objects < incoming.pack
index-pack verifies the pack checksum and asks Git to inspect the objects. unpack-objects materializes loose objects in the sandbox. Forge then uploads those objects under content-addressed repository keys, skipping keys that already exist. Object writes are also bounded to four concurrent operations; large individual objects use the existing multipart uploader instead of crossing the sandbox boundary as one long-lived in-memory buffer.
This separation matters for authority. The sandbox has Git and scratch space, but no object-store credential. The trusted runner Worker holds the R2 binding and copies only validated objects out of the sandbox. A malformed pack can consume temporary work, but it cannot move a branch.
It also corrected a second scaling mistake. A duplicate-heavy push should cost work proportional to the submitted pack, not to every object already stored in the repository. The current contract test sends an initial pack, sends it again, and then sends a mixed pack. The duplicate pass reports three objects examined and zero stored; the mixed pass stores only the three new objects without listing the repository object namespace.
Ref movement is a compare-and-swap
Object publication and ref publication are different claims. Writing a commit object does not authorize a branch change, and validating a pack does not prove that the branch still has the value the client observed before uploading.
For an existing ref, receive-pack uses the old SHA supplied by the Git client as a compare-and-swap condition:
UPDATE refs
SET sha = ?, updated_at = datetime('now')
WHERE repo_id = ? AND name = ? AND sha = ?
If another push has moved the branch, the statement changes no rows and Forge returns ref update mismatch. A migration-backed contract test exercises both stale updates and stale deletes and verifies that neither changes the ref. New refs use an insert that loses safely if a concurrent creator wins.
Before any create or update, Forge also confirms that the proposed target object exists in repository storage. The result is a useful failure invariant: upload failure, native validation failure, object publication failure, and stale-ref failure all leave the existing branch value unchanged.
This transaction is per ref. Forge processes the ref commands in a push one at a time, so a push that updates several refs is not an all-or-nothing transaction across the set. The implementation also does not re-walk the complete reachable graph before every ref update. Its guarantee is narrower: native Git validates the submitted pack, Forge publishes the unpacked objects, the target object must exist, and the old ref value must still match.
The 130 MiB result has a boundary
The recorded acceptance run exercised the public Git protocol with a clean 130.04 MiB push. Server acceptance took 85.73 seconds. A fresh clone of the resulting ref took 22.65 seconds, and the source and clone produced the same SHA-256 tree digest. The clone check matters because an HTTP success response alone does not prove that the stored repository can be read back intact.
Those timings describe one repository, one network path, and one run. Repository topology, compression, object reuse, and upload bandwidth all change the outcome. The raw timing transcript is not retained beside the article, so the numbers should be read as a historical acceptance observation rather than a reproducible benchmark or percentile.
The current production release gate proves a different and complementary claim. It creates a temporary repository-scoped credential, pushes a new canary branch, advances that branch with a second push, verifies both remote SHAs, clones the branch, runs git fsck --strict --no-dangling, and deletes the branch and credential. That canary is intentionally small. It verifies that the streaming path and existing-ref compare-and-swap still work after each release; it does not re-measure 130 MiB throughput.
Local contracts cover the heavier mechanism. The API-to-runner integration test stages real Git packs, validates them, checks duplicate reuse, and confirms temporary-pack cleanup. Runner tests cover malformed trees, bounded staged reads, upload retries, duplicate-heavy packs, large-object multipart transfer, and expired-stage cleanup. Together they support the safety design, not a universal performance claim.
Why not use a separate importer?
A background import service could accept larger archives and report progress through its own API. That is a reasonable option for histories above Forge's per-push envelope or for migrations that need resumable transfer. It lost as the default because it would make the common path less representative: a repository might import successfully but still fail its first ordinary push.
Streaming directly from the edge request into the native runner was another option. Forge retains that runner endpoint, but the production receive path first stages the pack durably. Staging adds object-storage traffic and a materialization step. In return, the edge does not have to keep a large pack in memory, and the runner works from an immutable input whose repository and request identities are checked at both boundaries.
The remaining limits are concrete. One pack must stay below 490,000,000 bytes and 500,000 objects. Larger histories need incremental pushes today. Pushes are not resumable across client retries, and multi-ref updates are not atomic as a group. Better phase timings and reuse for packs with many existing objects remain useful work.
Make publication the last operation
The reusable lesson is not specific to Git or Cloudflare. When a small mutable pointer exposes a large immutable data set, make pointer publication the final operation. Stage bounded input, validate it with the format's native tool, publish immutable content, and move the pointer only if its previous value still matches.
That ordering made a 130 MiB repository ordinary enough to use git push. A later release transition exposed the next durability gap: the staged pack survived upload, but cleanup erased the retry path between runners. The next meaningful performance test is not a larger number on a landing page. It is a repeatable size-and-topology matrix, retained with raw receipts, that shows where staging, native validation, object publication, and network transfer each become the limiting phase.
For the public deployment contract, including current ingest limits, see the Forge Deploy specification. Machine-readable integration guidance is available in llms.txt.
