SmolForgeField Notes

Notes on systems,
code, and craft.

All notes

Engineering

We Stopped Letting Feature Agents Deploy

Thirty-six handoffs and 3,481 coordination words made one release authority unavoidable.

A six-task rollout produced 3,481 words of cross-task coordination in 29 minutes; immutable candidates and one fenced controller moved production authority out of feature work.

The feature work was done. Six Forge tasks had clean candidates, passing checks, or both. Yet instead of finishing, they kept telling one another which SHA was current, whether a runner release was fenced, who had permission to advance main, which provider version was live, and when the production lane was clear.

In the busiest 28 minutes 43 seconds, the task logs recorded 36 explicit cross-task handoffs: one every 48 seconds. Two separate changes advanced main while releases were active, causing another round of freezes, provenance checks, receipts, and lane-clear messages. Across the six release-adjacent task logs we inspected, the wider eleven-hour period contained 90 such delegations.

The intervention was one sentence: feature agents no longer deploy. They may edit, test, preview, and produce an immutable candidate with evidence. One release controller alone may order candidates, acquire the production fence, mutate providers, verify the live version, and publish receipts. Three tasks completed shortly after that boundary was imposed; two stopped as clean candidates; one remained the release authority. Over the next day, that behavioral rule became a production queue with protected merges, durable release state, and a single fenced controller.

“You have become the human release coordinator you were supposed to eliminate.”

Operator intervention in the August 7, 2026 working thread.

The release agent's answer named the failure precisely: “I optimized the release path while still leaving a human-shaped mutex in the loop.”

Parallelism collapsed at production

The chart below is not an architecture sketch. Each block is one explicit delegation received by a release-adjacent task during the densest observed interval. Together, the 36 messages contained 3,481 coordination words—97 words per handoff on average. The timeline shows why message count alone understated the problem: two unexpected main changes triggered freezes and state reconstruction, then the same production receipts were broadcast repeatedly so every waiting task could rebuild its view of reality.

Chronological release-coordination storm: two unexpected main-branch changes force repeated freezes, state reconstruction, a fourteen-message reconciliation fan-out, candidate rebases, and an operator intervention. Thirty-six handoffs contain 3,481 coordination words in 29 minutes.
Coordination became the workload. Counted from six Codex JSONL session logs between 1:50:02 and 2:18:45 PM Pacific. Each block is one explicit <codex_delegation> delivered to one target task; word counts cover the message body. A message delivered to two tasks counts twice because two agents had to ingest and act on it. The final block is the audit intervention; thread identifiers and full message contents are omitted.

Some of these messages were necessary. An agent should stop when its expected source changes underneath an active release. It should verify the provider version rather than assume an upload is live. The failure was not that agents communicated; it was that every feature task had to participate in production serialization.

Parallel feature development had created a fan-in problem. Code could be written concurrently, but the last step touched shared mutable state: the canonical branch, the release lane, provider traffic, migrations, and production receipts. Every task that retained those capabilities became a potential release coordinator. The nominal release agent could advise them, but it could not eliminate the coordination because it did not exclusively own the mutation boundary.

The production ledger showed the same fan-out

The 36-handoff interval is a narrow view of the social cost. A separate read-only reconstruction followed all 54 terminal production-controller attempts from the first persisted v2 release through fence 56. Only 17 deployed: a 31.5% success rate. Summed across attempts, the controller recorded 13 hours 39 minutes—about four hours in queueing and validation, one hour waiting for the production fence, eight hours in promotion, verification, or rollback, and the remainder in unfenced supersession waits. These are aggregate attempt-hours, not a 13-hour critical path; some attempts overlapped and pre-enqueue branch CI is excluded.

Release width was the strongest warning in that window. All 13 attempts affecting eight or more components failed. During the later narrowed F44–F53 period, seven of ten attempts deployed. This does not prove component count was the only cause: policy defects, runner resets, network failures, and canary behavior differed between attempts. It does show why centralizing production authority was insufficient while one candidate could still rediscover every unrelated failure surface.

Waterfall of 54 production-controller attempts grouped into early bootstrap, policy reconciliation, full-platform fan-out, narrowed releases, and a final broad regression. Each bar separates queue and validation time, production-fence waiting, and active release time. About one in three attempts deployed, all 13 releases affecting eight or more components failed, and seven of ten attempts deployed in the later narrowed period.
One delivery incident contained 54 controller attempts. Pink is queue and validation, gray is production-fence waiting, blue ends deployed, and orange ends in another terminal state. The repeated late orange tails explain why broad releases generated more coordination: earlier component progress did not make the whole candidate releasable. Exact rounded totals, classifications, and limitations are available in the sanitized incident receipt.

The first visual was wrong in a useful way

The first diagram we drafted in the working thread projected each handoff onto the task that received it. It made the social topology unmistakable: Release, Sites, server rendering, Open Graph, Onboarding, Benchmark, and Landing kept pulling one another back into the same production conversation. The shaded bands show where main changed underneath active release work.

Exploratory handoff map with eight task lanes. Thirty-five operational messages jump repeatedly among Release, Sites, server rendering, Open Graph, Onboarding, Benchmark, and Landing before one operator intervention. Two shaded bands mark unexpected main changes.
Exploration 1 — map the conversation. This first visual exposed the many-to-many topology, but it made the sequence hard to read and gave every line equal weight. We kept it as a diagnostic artifact and replaced it as the primary evidence with the chronological timeline above, which connects the same interval to its two causal resets and reconciliation fan-out.

That failure sharpened the question. We did not need a prettier map of which agent messaged which other agent. We needed to identify the shared capability that forced those messages to exist.

Git and CI were innocent

The first instinct was to remove the machinery that seemed to create the waiting. Perhaps agents should abandon Git, write into one shared directory behind a live preview server, and skip CI until a batch deploy.

That would improve the editing experience, but it would not remove the production conflict. A shared workspace can hide branches and merges from feature agents. It can compose per-task overlays into one preview. It can run affected checks continuously instead of making every agent shepherd a pipeline. Those are worthwhile interface changes.

The release boundary still needs an immutable answer to “what are we deploying?” and a trusted answer to “what evidence applies to it?” Git commits happen to provide useful snapshots; CI happens to produce reusable proof. Neither requires every feature agent to move main, watch a rollout, or hold provider credentials.

Removing CI would delete evidence while preserving contention. Writing to one file would turn coordination into last-writer-wins. The system needed less shared authority, not less verification.

The next sketch moved the boxes away from individual agents and toward the authority boundary:

Three feature branches feed affected CI, which produces signed evidence and immutable artifacts. Below a production-authority boundary, a protected merge queue is the only writer of main and a single fenced controller is the only actor allowed to mutate affected Workers and edge before publishing one production manifest.
Exploration 2 — separate evidence from authority. Marker 1 keeps verification parallel and reusable. Marker 2 gives the protected queue sole authority to advance main. Marker 3 gives one leased, fenced controller sole authority to mutate providers. CI remained, but CI success stopped implying deployment permission.

This diagram exposed a naming mistake too. A deploy agent may inspect impact, select a compatible batch, or explain a failure. It must not become a second production writer. The stateful mutation boundary belongs to deterministic controller code with a durable lease and receipts.

The capability boundary

The new contract separates makers from the release owner.

Feature agents may:

  • edit their scoped workspace and use a live composite preview;
  • run affected tests, builds, and policy checks;
  • freeze an immutable candidate with source and artifact digests;
  • attach validation evidence and declare dependencies or migration needs; and
  • observe queue, release, and production status.

Feature agents may not:

  • advance the canonical production source;
  • activate provider versions or change traffic;
  • apply production migrations;
  • overwrite another candidate's release state; or
  • declare themselves deployed from a green build or healthy preview.

The release controller owns those mutations. It accepts candidates, orders or batches compatible changes, rejects stale evidence, acquires an environment-scoped lease, promotes affected components in dependency order, proves the exact public version, and emits one receipt per candidate. If it crashes or loses its lease, a monotonic fencing token makes its older authority permanently stale.

This is a capability boundary, not a job title. A “DevOps agent” that merely watches another deploy agent still leaves two actors negotiating production. Monitoring, analytics, and policy can advise the controller; only the fenced controller can mutate the release.

One rule changed the system

The first version of the fix was behavioral, not software. The active tasks were told to stop pushing and deploying, preserve exact candidate SHAs, and hand production responsibility to one task.

The immediate observation was striking. Onboarding, benchmark/blog, and Sites completed soon afterward. Open Graph and server-rendering work stopped thrashing and remained as clean, verified candidates. The release task became the only production authority. Those queued candidates later shipped through the protected path; the landing-page release followed, and server rendering reached all document routes without restoring deployment authority to the feature tasks.

That is not a controlled before-and-after benchmark. The tasks differed in scope and some were already close to completion. It does show that the same code, CI, Git repository, and agents behaved differently when one permission changed. Coordination messages had been generated by the operating contract, not by an intrinsic inability to work in parallel.

Forge then made that rule enforceable. Candidates enter a durable merge queue. Release state lives in D1. An environment-scoped Durable Object owns the lease and fencing token. Provider executors must present the current fence at mutation boundaries. Direct provider access is being reduced to audited break-glass recovery instead of a convenient second lane.

Make Git disappear, not snapshots

The best agent experience may still look like the proposed shared codebase. A feature agent opens a logical workspace, edits files, and sees its changes in a live preview composed with other compatible candidates. It does not manage branches, rebase release trains, or wait beside a green CI screen.

Underneath, the platform preserves isolation. Each task writes to an overlay or change set rather than destructively sharing one filesystem. A candidate freezes the overlay against an exact base. Continuous validation keys evidence to the candidate's inputs. The release controller chooses a compatible set and materializes one immutable release snapshot.

Git can become an implementation detail of that snapshot system. CI can become a continuously updated evidence service. Both may disappear from the feature agent's conversational surface without disappearing from the safety model.

This distinction matters because “one shared file” solves the visible ceremony by deleting isolation. The scalable abstraction is one shared preview, many isolated changes, and one serialized production authority.

The third sketch expanded that boundary into the full development loop the thread was reaching for:

Three feature agents write isolated overlays into one live integration workspace. The workspace produces a continuous composite preview and an immutable frozen batch. A deploy agent proposes the batch, a deterministic controller mutates production, and a read-only DevOps observer returns reliability recommendations as ordinary changes.
Exploration 3 — hide Git behind a live integration workspace. The shared object is a continuously composed preview, not a writable filesystem. Each agent retains an isolated overlay. The deploy agent proposes; the controller mutates. The DevOps observer diagnoses production but can only return a normal, reviewable change to the workspace.

This is where “abandon Git” became more precise: abandon branches, pushes, and rebases as the feature agent's conversational interface. Keep content-addressed snapshots, diffs, rollback, and evidence underneath. The platform can feel live without making release state mutable.

Close the SDLC loop with evidence

Once production mutation is singular, the surrounding agent roles become clearer. Feature agents turn bets into candidates. The release controller turns compatible candidates into one verified production state. Observability and user-feedback agents inspect production behavior, support signals, experiments, and usage data, then propose the next evidence-backed bets.

That final agent should not silently create roadmap authority any more than a feature agent should silently create deployment authority. It can identify patterns, quantify pain, and draft feature proposals. A product owner or explicit policy decides what enters the build queue.

The final thread sketch originally connected product intelligence directly to feature agents. Annotating it revealed the same authority error one layer earlier: an analytics agent that can commission its own ideas is a roadmap coordinator in disguise. The production version adds an explicit product owner or policy gate between synthesis and implementation.

Production signals flow to a product intelligence agent, then through an explicit product owner or policy gate to feature agents, live integration, a batch release agent, and a fenced release controller. Verified outcomes return as the next production signals.
Exploration 4 — close a governed product loop. Marker 1 is read-mostly synthesis: turn usage, errors, support, experiments, and retention into evidence-backed bets. Marker 2 is roadmap authority: prioritize, constrain, or reject those bets. Marker 3 is release authority: promote one exact candidate set or roll it back. Outcomes become new signals rather than instructions sent through chat.

The resulting loop is not “three agents model the whole SDLC.” It is a chain of bounded capabilities:

production evidence → prioritized bet → isolated change + live preview
→ immutable candidate + proof → fenced release → production evidence

Every arrow carries a durable artifact. No role needs a chat message to reconstruct whether another role already acted.

What the first production queue proved

The 36-handoff interval is intentionally narrow. It counts explicit task-to-task delegations received by six selected release-adjacent logs, not every coordination message in the system. The broader 90-message count covers eleven hours and the same six target logs. Necessary safety stops are included because they still consumed agent attention. This was an operational incident, not a randomized experiment.

The controller is no longer shadow-only. It has accepted production candidates, protected main, reused component evidence, fenced provider work, resumed interrupted releases, and carried the Open Graph, landing, and server-rendering changes to live traffic. The first CI fast-path candidate also exposed a stale post-merge validation event after its branch evidence had already been accepted. Forge removed that legacy event path in the next queued change rather than treating the architecture as finished.

The remaining proof is longitudinal: record candidate-ready time, queue time, controller work, release duration, supersession, and post-release defects for several parallel feature cohorts. The success metric is not zero communication or one fast canary. It is that feature agents finish when their immutable candidate and evidence are accepted, while one fenced controller absorbs the irreducible production serialization.

The normative authority, state, fencing, retry, and break-glass rules live in the Forge release state machine v2. The lesson from this rollout is already durable: parallelize making, centralize releasing, and let evidence—not agents negotiating in a chat—cross the boundary.