A push failed with Git's bluntest server-side verdict: remote unpack failed. Retrying from a clean clone did not help. Disabling thin packs did not help. Creating a new branch that pointed at an object already present on the remote—and therefore had no meaningful new object data to contribute—failed at the same boundary. The local commits remained intact, but the remote branch was never created.

That result was especially uncomfortable because Forge had already rebuilt this path around durable staging. The pack crossed the edge request into object storage before a trusted runner validated it. A release canary pushed, advanced, cloned, and checked a repository after deployment. This was supposed to be the hardened design.

The bytes were durable. The operation was not. A runner release could reject the handoff, and the repository Worker deleted the staged pack before it had received a successful ingest receipt. The repair was to give the original receive operation a retry owner, retain the immutable stage through transient release states, and keep ref publication as the final step.

Where unpack failed sits in a push

Git's message makes more sense when placed in the receive-pack lifecycle. Before a push, the client discovers the remote refs and capabilities. It then sends a command list containing each proposed ref change: the old object ID it observed, the new object ID it wants, and the ref name. If the remote lacks required objects, the command section is followed by a pack.

The server has several jobs after that request arrives:

1. advertise refs and capabilities
2. authenticate and authorize the proposed ref commands
3. receive the pack, if the update needs object data
4. validate its checksum, objects, and deltas with native Git
5. publish the immutable objects
6. compare-and-swap each ref from its expected old ID to its new ID
7. report `unpack ok`, then `ok` or `ng` for each ref
8. schedule workflows and other consequences of successful ref changes

unpack failed is the protocol's report for a failure before ref publication, not a diagnosis of corrupt compression. A malformed pack can cause it. So can unavailable storage, a failed trusted runner, or—as here—a release handoff that never produced a receipt. Forge correctly left the ref unchanged, but collapsed several possible causes into the same client-visible boundary.

The zero-object experiment was decisive because it removed the most tempting explanations. The proposed branch pointed at a commit the server already had. Its protocol request could still contain an empty pack envelope, but changing compression, delta reuse, repository cleanliness, or payload size could not repair a control-plane admission failure.

Durable staging solved only the first boundary

Forge's earlier large-push redesign moved pack ownership out of the edge request. The repository Worker parses the bounded command prefix, writes the remaining stream to a repository-scoped multipart object, and asks the private runner to process that immutable staging key. Native Git verifies the pack. The trusted runner publishes content-addressed objects. Only then may the repository Worker move a ref.

That architecture fixed a real scaling problem. The request no longer buffered and expanded a migration-sized pack in JavaScript, and a branch could not expose objects that had not passed native validation. But the first implementation made one subtle promise it did not keep.

The design note said staged packs would be deleted after the runner receipt. The implementation put deletion in an unconditional finally block around one runner request. Success returned a receipt and deleted the stage. A transient 503 also deleted the stage. Object storage made the input durable across process boundaries, but no component owned retry after the first handoff failed.

This is a common distributed-systems trap: a durable artifact is not the same thing as a durable operation. The operation also needs an identity, a retry owner, a completion receipt, and cleanup tied to a terminal outcome.

Release draining exposed the missing owner

The coupling entered when native Git ingest first moved to the runner. Its endpoint reused the runner's general admission predicate. That seemed conservative: if a runner should not start new work, it should reject new requests.

Later, runner releases gained a real lifecycle. The incumbent enters draining so long CI and Workspace jobs stop starting. A successor enters verifying while Forge checks its exact version and toolchain. Only the verified successor becomes ready. Enforcing that lifecycle turned the old shared predicate into a reachable failure window.

Git ingest is not the same class of work as a long build. It is request-scoped, keyed by immutable staged input, and safe to retry before refs move. The already-verified incumbent can finish it while draining. The verifying successor must stay closed, but that should defer the handoff rather than destroy its input.

The incident was therefore not pack corruption and not a missing upload limit. One admission rule was serving two operations with different retry and lifetime semantics.

The old cleanup erased the retry path

The before-and-after sequence shows why the R2 stage alone was insufficient:

BEFORE
client → repository Worker → stage pack in R2
                         ↘ runner handoff → transient 503
                           finally: delete stage
client ← unpack failed     ref remains unchanged

AFTER
client → repository Worker → stage pack in R2
                         ↘ handoff → transient 503
                           retain and retry the same staging key
                         ↘ runner receipt
                           compare-and-swap ref → delete stage
client ← unpack ok

The repair separates Git-ingest admission from long-job admission. A matching draining runner remains eligible for ingest. Transport errors, 429s, and 5xx responses are retried within a bounded 35-minute handoff window using the same immutable staging key. The stage is deleted after success or after that bounded operation reaches a terminal failure, not after the first attempt.

The ref rule did not change. Forge still checks the runner's receipt against the staged byte count and publishes required objects before comparing the old ref value with the value supplied by the client. Retries can repeat ingest work; they cannot move a ref early.

The canary proved states, not transitions

The existing production release gate was useful. It created a temporary credential, pushed a branch, advanced it, checked the remote SHA, cloned it, ran git fsck, and cleaned up. That proved the public protocol and object/ref ordering on an active release.

It did not keep a push alive while the runner moved through its release states:

runner:     ready ───── draining ───── verifying ───── ready
old proof:    ●                                         ●
missing:              one receive-pack spanning the transition

A canary before and after a rollout samples two stable points. It can miss a bug that exists only between them. This is the same distinction as unit-testing each state in a state machine without testing the transitions that connect them.

The stronger release proof is adversarial: begin one public push, move the runner through draining and successor verification while that push is active, then require the original client operation to finish with the intended ref and a readable clone. Until that transition canary runs in production, the code and local contracts prove the mechanism, not the absence of every rollout failure.

Production closeout: the repaired stable state held

Forge later closed the original incident against audited source dcf25e31a72b94c9f8bd10609e8d1b2cf67328b0. A repository-scoped production canary created a new branch, advanced its existing ref, cloned it into a clean directory, ran strict fsck, deleted the temporary branch, and revoked its credential. The explicit zero-object case reported Total 0, the server advertised no-thin, native ingest indexed zero objects, and the ref still moved successfully. Changing pack shape was no longer able to reproduce the admission failure.

The final Forge mirror matched all five advertised refs and contained 13,463 reachable objects in a 59.7 MB pack. Its small clean canary passed strict fsck without exceptions. The historical Forge repository itself retained one pre-existing treeNotSorted object from an older commit, so the full mirror check ignored that single documented warning rather than mislabeling it as new transport corruption. Snapshot generation 580 then completed on its first attempt, after the temporary verification refs were gone.

That evidence closes the stable-release failure reported here. It does not replace the transition canary described above: these pushes ran against an already active release, not through a live ready → draining → verifying → ready handoff. The broader transport measurements are in Forge Git Is Fast. Cold Checkout Isn’t., while the remaining release-authority work is tracked in the release modernization incident synthesis.

unpack failed is a boundary, not a cause

The first operational improvement is diagnostic. Forge now attaches a request identifier to unpack failed and records the failed ingest phase and bounded runner detail. Operators should inspect that evidence before asking a user to rebuild a pack. Public incident identifiers remain out of this article, but the distinction they expose belongs in the product contract.

The second improvement is semantic: transient infrastructure must remain inside the server's retry boundary. Terminal native validation or storage failures can still produce unpack failed; pretending otherwise would weaken the protocol. A normal runner release must not.

The durable lesson is concise: make the operation durable, not only its bytes. Stage immutable input, assign one owner to retry it, retain it until a trustworthy receipt or bounded terminal outcome, publish mutable refs last, and test the lifecycle transition that threatens those guarantees.

Current failure meanings and operator actions are documented in Forge Deploy failure diagnostics. The public receive limits and ref-publication contract remain in the Forge Deploy specification.