SmolForgeField Notes

Notes on systems,
code, and craft.

All notes

Architecture

High-Throughput Agents Need a Routed Monorepo

One live change environment freezes into a routed transaction.

A 54-attempt release incident shows why high-throughput coding needs routed changes, independent component releases, and bounded release authority—not less CI.

Forge is a self-hosted Git, continuous integration (CI), and deployment platform that also hosts its own codebase. In one experiment, six Codex coding threads worked simultaneously in isolated Git workspaces. Their assignments ranged from authentication and React rendering to onboarding and release-system changes.

All six threads stopped talking mainly about their assigned features and started reconstructing rollout state: which source was canonical, who owned production, whether another deployment was running, and which commit users were seeing. When we removed deployment authority from the feature agents, three tasks finished quickly, two became clean queued candidates, and one release task retained authority. Coding throughput had increased; delivery throughput had not.

Comparison of today's delivery workflow with a proposed Routed Monorepo. Six feature agents each repeat code, push, CI, preview, feedback, and repush before converging on one global merge and deploy queue. In the future proposal, persistent Change Environments freeze into Routed Changes, a Program Database and verified graph route work into parallel web, authentication, build, and schema release lanes, a deterministic release controller promotes valid work, and product feedback returns from production. Git remains the existing history substrate.
Today, every feature agent repeats the CI and preview loop before waiting on one global deployment queue. The proposed Routed Monorepo turns development, evidence, integration, and release into explicit colored primitives, so only work that shares a real production resource must serialize. Open the diagram for the full-size version.

We do not need to replace Git to start fixing this. We need a Routed Monorepo: a repository that can map every immutable change to affected components, required evidence, a composed preview, and production resource locks. Git can remain the source record while the routed repository becomes the delivery model. Over time, that model points toward something more ambitious: a transactional Program Database.

We scaled execution, not delivery

Agent orchestrators are becoming good at the execution plane. OpenAI's Symphony specification describes the important primitives: poll a tracker, claim an issue, create an isolated workspace, supervise a persistent coding session, reconcile state, and retry failures. It even leaves Git and workspace population implementation-defined.

An isolated workspace prevents two agents from overwriting the same checkout. It does not tell either agent that both changed the authentication contract, selected the same database-migration number, or need to update the same public web service. Ten independently successful runs can still produce one large integration queue.

That scheduler boundary is useful, but it is not the whole software development lifecycle. Forge stored source in branches, intent in tickets, results in CI, and live state in its deployment system. Agents reconstructed the links in conversation.

We inspected local JSONL logs for four Codex sessions: authentication, server rendering, onboarding, and release control. An event can be a message, tool call, result, or status—not elapsed work. The traces ranged from 793 to 6,319 events. The 18.9 MB release-control trace contained six logical jobs plus deployment results. The counts are not a performance benchmark; they show that one conversation was neither one change nor one release.

The release ledger made the thrash visible

We later reconstructed the first complete release-platform incident window from terminal production-controller records instead of chat recollection. It contained 54 attempts across roughly 28 hours. About one in three deployed. Summed across every attempt, the controller spent about 14 hours: roughly four in queueing and validation, one waiting for the production fence, eight actively promoting, verifying, or rolling back, and the remainder in superseded waits and bookkeeping. These are aggregate attempt-hours, not wall-clock duration; some attempts overlapped, and pre-enqueue branch CI is outside the count. The public diagrams round the totals and omit internal identifiers.

Waterfall of 54 production-controller attempts grouped into early bootstrap, policy reconciliation, full-platform fan-out, narrowed releases, and a final broad regression. Each horizontal bar separates queue and validation time, production-fence waiting, and active release time, with terminal success or failure. About one in three attempts deployed; all 13 releases affecting eight or more components failed; a later narrowed period deployed seven of ten attempts.
The release waterfall shows a delivery incident, not one slow deploy. Wide candidates were the strongest warning: all 13 attempts affecting eight or more components failed, while seven of ten attempts deployed during the later narrowed period. Queueing mattered, but active promotion, verification, and rollback was the largest time bucket.

That changes the diagnosis. Removing CI would have saved only part of the delay while deleting the evidence boundary we needed most. The sharper predictor was release width: late failures wasted all of the earlier serial work, and a broad candidate gave more components a chance to become that late failure. Narrowing releases improved outcomes, but only temporarily, because the operating process still allowed the next repair to fan out again.

The human experience also ran on a different clock from the controller. A 7-hour-23-minute turn mixed account recovery, database repair, production coordination, release-system changes, and 13 controller attempts. Another 6-hour-57-minute turn bundled five requested batches, live adoption, and 18 attempts. A later 2-hour-53-minute turn expanded three architecture requests into six attempts. Those durations were real, but they were scope containers, not measurements of CI or a single deployment.

A 28-hour timeline compares six long agent turns with 54 shorter production-controller attempts arranged in four lanes. The agent clock includes turns lasting 7 hours 23 minutes, 6 hours 57 minutes, and 2 hours 53 minutes, while the machine clock shows many separate deployed and non-successful attempts. Three callouts explain that long turns were scope containers, failures authorized more implementation, and the missing boundary was a stop condition.
The two clocks explain why the experience felt like “a seven-hour release.” The release turn kept absorbing recovery, architecture, coordination, and retries. Measuring only controller latency would miss the organizational failure; blaming the whole duration on one deploy would misdiagnose it.

The recurring loop was simple. A feature or repair merged, impact fanned out, components ran serially, and a late component failed. Rollback left partial receipts and mixed expectations. Instead of restoring a known baseline and ending the release turn, the failure became permission to repair the platform inline. Main advanced, the candidate broadened, coordination expanded, and the loop began again.

An eight-step feedback loop moves from a merged feature or repair through impact fan-out, serial work, late component failure, rollback, new repair scope, another advance of main, and expanded coordination before repeating. A green stop-loss panel adds bounded release authority: a fixed candidate, one retry budget, a circuit breaker for repeated causes, and a separate Routed Change for the repair.
Routing is necessary but insufficient. The controller must also prevent a release failure from manufacturing the next change. Platform repair belongs in a separate Routed Change with its own review and release, not inside the failing release turn.

The missing primitive is bounded release authority: one fixed candidate, one retry only when the inputs are unchanged and the failure is explicitly retryable, then a circuit breaker when the same cause repeats. The controller restores the stable baseline and enters incident mode. Any platform repair becomes a new Routed Change. This is the stop-loss that turns routing from a map of the blast radius into control over it.

Isolation exposed the next bottleneck

The clearest pressure test involved three ordinary product changes. One agent was replacing Forge's browser-session code with Better Auth, an authentication library. A second was adding server-side rendering (SSR) to the public React application. A third was adding progress milestones to the first-run onboarding interface.

Workspace isolation worked. The authentication agent reached a clean commit without interrupting the server-rendering agent. The onboarding agent completed and shipped from its own checkout. This is exactly the concurrency an execution system should provide.

But the finished changes were not independent. Authentication and SSR both changed package-lock.json, packages/web/package.json, App.tsx, useAuth.tsx, and LoginPage.tsx, then required the same public web deployment. More importantly, the rendering change kept authenticated pages in the browser, while the authentication change made sessions readable on the server. Should the server now render private data? A text merge could not make that security and product decision.

The release-system agent and authentication agent also created different SQL migration files with the same ordered prefix, 0009. Each workspace saw 0009 as the next available number. Neither agent had behaved irrationally; the shared migration sequence had changed outside its isolated view.

Git successfully isolated the writers and preserved their work. What was missing was a machine-readable delivery contract that could discover those shared assumptions before release.

A monorepo that routes change

A traditional monorepo answers: where is all the code? A Routed Monorepo answers: if this commit changes, what becomes invalid, and which valid parts can still proceed?

For the first version, I would require a physical monorepo. Source, build definitions, shared contracts, migrations, deployment topology, and policy need one verifiable root. Federation can come later if it preserves the same guarantees.

This borrows from microservices and microfrontends without requiring either architecture. Microservices made runtime and deployment boundaries explicit. Microfrontends applied independent delivery to user-interface surfaces. A Routed Monorepo makes change boundaries explicit: which source, contracts, tests, artifacts, and production resources must move together.

We should not create a distributed system merely to let agents work in parallel. A modular monolith can expose independently releasable components. A repository full of microservices can still have one global queue if its boundaries are only conventions. The graph, not the process topology, supplies the guarantee.

The repository must provide four things:

  1. A verified component graph. Every source and configuration input maps to a releasable component—such as a web app, authentication service, build runner, or database schema—plus its dependencies, tests, artifact builder, deployment target, and production resources. Declared ownership is checked against actual imports and bundle inputs.
  2. An immutable Routed Change. Freezing a live change environment produces one candidate with an exact Git source state, intent, affected components, evidence, required resources, and a provenance link.
  3. A disposable composed preview. The system can combine candidate changes without mutating their source branches, expose the result at an immutable URL, and report file, component, schema, and explicit decision conflicts.
  4. Resource-scoped, bounded release control. Independent component releases proceed concurrently. Only changes that require the same production resource or shared invariant serialize. One release cannot change its own candidate, and a repeated failure opens the circuit instead of authorizing an inline platform rewrite.

That is enough to test the thesis. It does not require a new filesystem, semantic refactoring language, product-analytics agent, or replacement for Git history.

One workspace, three live interfaces, one freeze boundary

Each feature agent should receive one persistent Change Environment. Its development shell, incremental test and review agents, and stable preview URL all observe the same mutable state. Feedback does not require a push, a clean remote checkout, and a complete reconstruction of context. A reviewer can open the real dev server while the feature agent changes it.

Development, testing, and preview should share a substrate, but they should not share a trust level. Continuous tests give fast feedback inside the environment. Freezing the environment creates immutable source inputs. A trusted verifier then attests those exact inputs independently before production policy accepts the evidence. CI does not disappear; it moves from a distant development loop to an attestation boundary.

This gives the lifecycle one sharp transition:

mutable Change Environment
  = development shell + continuous tests/review + stable preview
                              │ freeze
                              ▼
immutable Routed Change
  = exact source + intent + evidence + resources + provenance

The atomic delivery unit is a Routed Change

The atomic source unit remains a Git commit. The new atomic delivery unit is a Routed Change: a small immutable envelope around that commit.

change: auth/new-sessions
source_sha: 569cf181

intent:
  migrate browser sessions to server-readable sessions

affected_components:
  - auth-service
  - web-app

evidence:
  auth-service: receipt://auth-tests
  web-app: receipt://web-build

resources:
  - database/migrations
  - production/auth-service
  - production/web-app

provenance:
  thread: thread://auth-change

The agent develops this manifest while working; freezing makes it immutable. Deterministic tooling verifies changed paths, the dependency set, evidence inputs, and resources. The transcript remains available when a reviewer needs a design decision, but downstream systems do not ingest megabytes of reasoning by default.

Most cross-agent messages also collapse into structured state. “Is the web app still deploying?” becomes a ledger read. “Migration 0008 is reserved” becomes a resource reservation. “Do not deploy this branch” becomes an authority boundary.

One free-form callback remains useful:

needs_decision:
  subject: authenticated SSR
  question: may server rendering consume authenticated sessions?
  requested_from: thread://auth-change
  blocks:
    - composed-preview
    - production/web-app

The source agent answers the narrow question or updates its Routed Change. We do not need a vocabulary of agent roles and protocol verbs before this simpler mechanism proves insufficient.

Integration and release should remain separate authorities even if one service implements both at first. An integration agent composes candidates, classifies conflicts, and can wake a source agent; it does not hold production credentials. The release controller accepts only frozen candidates and trusted evidence, acquires production locks, and updates the ledger. “Release agent” and “release controller” name the same authority here—the latter emphasizes that its critical behavior should be a deterministic state machine, not another agent supervising an agent.

Route real conflicts, not every change

The impact router computes the smallest sufficient validation and release plan from the component graph and Routed Change. Forge's authority-scoped Worker split is the concrete component inventory this kind of routing requires.

The authentication change illustrates the boundary. Its service can build while server rendering waits for a decision. Unrelated source-storage, build-runner, content, and notification components remain available to other changes. But its ordered SQL migration must follow the release-system migration, and both authentication and server rendering need to activate the same public web Worker.

The system therefore reserves database/migrations and production/web-app. It does not acquire a global production lock.

The release controller freezes the candidate, verifies its exact inputs, and acquires resource-scoped fencing tokens—monotonic lock numbers that reject stale deployers. It promotes immutable artifacts, checks health, and records rollback. The production ledger records active web, authentication, build-runner, and schema versions independently.

Its authority ends with that frozen candidate. It may retry once with identical inputs when policy classifies the failure as retryable. A repeated cause restores the known stable baseline and opens an incident; it does not wake a coding loop inside the production lane. The repair is reviewed, frozen, and routed as a separate change. Self-deploying release infrastructure also needs an external or last-known-good executor, so a candidate cannot replace the machinery still judging it.

This is also where CI changes shape. We do not remove it. The graph routes each change to the checks that prove its affected closure. Evidence is reusable only when its source, dependencies, toolchain, and policy inputs still match. A global gate remains available for a real global invariant, not as the default expression of caution.

The Routed Monorepo points to a Program Database

The Routed Monorepo is the practical migration path. The Program Database is the end state.

Git is already a database, but its native records are files, trees, commits, and refs. An agent-native system must also query program structure, contracts, intent, evidence, resources, and live state. It should answer “what becomes invalid if this interface changes?” without reconstructing the answer from several tools and conversations.

The database analogy is operational, not cosmetic:

Current workflow Program Database model
Commit and diff Immutable source record
Branch or workspace Mutable snapshot
Routed Change Transaction envelope
Component graph Schema and dependency indexes
CI checks Constraints and derived evidence
Composed preview Materialized view of candidate transactions
Release resource locks Transaction locks and fencing tokens
Production ledger Committed live state
Agent transcript Transaction provenance, not the transaction

In that model, an agent conversation is closer to a query session and transaction log than an atomic change. The agent reads state, proposes operations, records decisions, and produces a compact transaction. A merge or release agent reads that transaction first and opens the provenance only when a decision is ambiguous. If necessary, it sends one typed needs_decision callback that wakes the source agent instead of absorbing its entire transcript into every downstream context window.

Git can remain the durable source interchange and audit format while this model matures. Later, semantic program operations could compose directly and Git trees could become materialized views of committed program state. That future is worth naming because it explains the direction of the smaller design:

Microservices parallelized deployment by splitting the application. A Program Database should parallelize delivery without forcing us to split the application.

Build five things, then measure

The first prototype needs only the five capabilities above: persistent Change Environments, exhaustive component ownership, Routed Change manifests, disposable composed previews, and resource-scoped release locks. Symphony or another orchestrator can continue to own issue scheduling, workspace lifecycle, retries, and agent observability.

We can replay the authentication and server-rendering changes through the prototype and ask concrete questions:

  • Did it identify the five shared files before either agent finished?
  • Did the component graph catch the authentication runtime boundary?
  • Did it reserve a migration identity instead of allowing two 0009s?
  • Did it request one decision about authenticated SSR?
  • Could unrelated components continue while the web app waited?
  • Could an operator read the exact active versions without messaging an agent?

If those answers are yes, the next test is sustained load. Increase feature-agent concurrency and measure completed production changes, queue time, rollback rate, and human interventions against today's branch-and-global-gate workflow.

Several larger ideas can wait. Semantic program operations may eventually compose better than text patches. Transcript compilers may derive Routed Changes automatically. Production analytics may generate the next work item. None is required to test the observed bottleneck or to start accumulating the records a Program Database would need.

The immediate claim is smaller and more practical than “Git will disappear”: high-throughput coding needs a monorepo that can route each frozen change through integration and production without making unrelated changes wait. The larger wager is that, once those routes become structured and queryable, the repository stops behaving like a folder of files and starts behaving like a database of the program.