# SmolForge by swyx A fully-realized GitHub clone built entirely on Cloudflare infrastructure. Push code, browse files, manage issues, create pull requests — all backed by Workers, D1, R2, and Durable Objects. **Live:** [forge.smol.ai](https://forge.smol.ai) · **Engineering blog:** [forge.smol.ai/blog](https://forge.smol.ai/blog) · **RSS:** [feed.xml](https://forge.smol.ai/blog/feed.xml) ## What This Is SmolForge implements the core GitHub experience from scratch: user auth, repository CRUD, independent repository forks, Gist-style multi-file snippets, the Git Smart HTTP protocol (push/clone/fetch with a real `git` CLI), file browsing, branch management, issues, pull requests, collaborators, and an organization/team permission system. Forge is now a routed Cloudflare system rather than a single Worker. Twelve release components separate the public edge, identity, content control, repository data plane, deployment control plane, notifications, build runner, hosted-site edge, deployed-application runtime, wiki, AI routing, and Slack agents. They share explicit contracts and are built, tested, and released according to component impact. Forge began in a single session using Claude in Cowork mode, with early scaffolding help from [ralph-claude-code](https://github.com/frankbria/ralph-claude-code) for autonomous development loops. The system has since grown into the component architecture below. ## Architecture ```text forge.smol.ai | +-------v-------+ | public edge | | web/blog/docs | +-------+-------+ | +------------+------------+------------------+ | | | | +------v------+ +---v--------+ +--v----------+ +-----v--------+ | identity | | content | | repository | | deploy-control| | auth/account| | gists/cards| | Git + repos | | projects/releases | +-------------+ +------------+ +------+------+ +------+-------+ | | +--------v--------+ +--------v--------+ | runner / wiki / | | deploy-runtime / | | notifications | | sites-edge / AI | +-----------------+ +-----------------+ control D1 + domain D1s · R2 Git/artifacts · Durable Objects ``` The component graph is declared in [`config/forge-components.json`](config/forge-components.json). Identity, repository, Deploy control, and notifications have independent workspace entrypoints and Wrangler contracts; `packages/api` is their internal implementation kernel while feature code is moved to its final owner. Changes are routed to affected validation suites and runtime components, with a fail-closed full gate when ownership cannot be established. **Cloudflare D1** stores relational state behind explicit domain ownership. The current product/control-plane source is `cloudforge-next`; an executable inventory in [`config/forge-database-domains.json`](config/forge-database-domains.json) maps every canonical table to its future database and reports cross-domain foreign-key blockers. Domain databases are additive shadow targets until schema, row-count, and content-digest parity pass and the owning runtime has no cross-domain transaction. The AI router already owns its reservation and immutable invocation ledger in `cloudforge-ai-usage`. **Cloudflare R2** stores Git objects, packfiles, release artifacts, and other immutable blobs. Git objects are keyed by repository and object SHA. **Durable Objects** provide repository coordination and the production release lease. The release lease issues a monotonic fencing token; stale controllers cannot mutate providers. **Hono** remains the Workers-native HTTP framework for the API components. ## Tech Stack | Layer | Technology | Why | |-------|-----------|-----| | Runtime | Cloudflare Workers | Edge compute, zero cold starts, global deployment | | Database | Cloudflare D1 (SQLite) | Serverless relational DB at the edge | | Object Storage | Cloudflare R2 | S3-compatible, no egress fees, stores git objects | | Concurrency | Durable Objects | Repo locks plus fenced production release authority | | API Framework | Hono | Lightweight, Workers-native, great DX | | Auth | JWT (HMAC-SHA256) + PBKDF2 | Stateless auth via Web Crypto API | | Frontend | React 19 + React Router SSR + Vite + TailwindCSS | Edge-rendered public routes and hydrated product UI | | Build/Deploy | Forge release v2 + Wrangler | Affected CI, merge queue, fenced promotion, immutable provider versions | | Testing | Vitest | Fast, TypeScript-native test runner | | Monorepo | npm workspaces | Simple, no extra tooling needed | ## Git Protocol Implementation The most interesting part. SmolForge implements the [Git Smart HTTP Protocol](https://www.git-scm.com/docs/http-protocol) from scratch: **Endpoints:** - `GET /:owner/:repo.git/info/refs?service=git-upload-pack` — Reference discovery (clone/fetch) - `GET /:owner/:repo.git/info/refs?service=git-receive-pack` — Reference discovery (push) - `POST /:owner/:repo.git/git-receive-pack` — Receive pushed objects - `POST /:owner/:repo.git/git-upload-pack` — Send objects for clone/fetch **What's implemented:** - Full packfile parsing with zlib decompression (RFC 1950) - OFS_DELTA and REF_DELTA resolution for delta-compressed objects - Custom Huffman decoder for scanning deflate streams without full decompression - Packfile generation with proper SHA1 checksums - Object graph walking for `git clone` (collects all reachable objects) - Stateless fetch negotiation with `multi_ack_detailed`, `no-done`, and immediate `ACK ready` pack streaming for incremental pulls - pkt-line protocol encoding/decoding - HTTP Basic Auth for git operations The packfile parser (`packages/api/src/git/packfile.ts`) includes a hand-written deflate stream scanner that traverses the block structure (stored, fixed Huffman, dynamic Huffman) to find object boundaries without decompressing — needed because packfile entries are concatenated zlib streams with no length prefixes. ## Product Implementation Details The landing page intentionally keeps implementation details light. This section is the deeper technical reference for how the product surface is built. ### Frontend and repo browsing - React 18, React Router v6, Vite, and Tailwind CSS power the web UI. - The file browser includes a VS Code-style tree, breadcrumbs, syntax highlighting, filetype icons, hover prefetching, and inline file editing with a commit panel. - Markdown rendering includes heading anchors, generated table of contents, copy buttons on code blocks, scrollable code blocks, table rendering, and dark mode support. - Accessibility options include light/dark/system themes, high-contrast mode, configurable font sizes, and reduced motion. ### Git and repository operations - Git Smart HTTP support covers clone, fetch, push, refs, and packfile exchange. - Push operations use repo-level coordination to avoid concurrent write races. - Repositories support `public`, `unlisted`, and `private` visibility. Unlisted repositories are readable by direct URL but excluded from public profiles and discovery. - Forking runs as a durable, resumable repository job with byte/object progress, bounded copy concurrency, three attempts, cancellation, and cleanup. The destination remains private until every Git object and snapshotted ref is present. It keeps source attribution and licensing while excluding issues, collaborators, secrets, workflows, webhooks, and transcripts. - Repository headers display SPDX license identifiers. The curated one-click choices are MIT, Apache-2.0, and AGPL-3.0-or-later; adopting one commits canonical SPDX license text as `LICENSE`, including for a repository's initial commit. - Pull requests include branch comparisons, commit lists, changed-file summaries, merge status, and permission-controlled fast-forward merges with stale-head/base protection. - Repository collaboration includes collaborators, organizations, teams, role hierarchy, and team-repo permissions. ### CI/CD Actions - Workflow configs, runs, jobs, steps, logs, artifacts, and secrets are modeled in the database. - Workflows can run on push, pull request, or manual triggers. - Container jobs support manager-aware `cache: auto`, `cache: npm` for download/tool caches, and `cache: npm-exact` for dependency snapshots. Generated Deploy builds choose `auto`; every cache identity includes the normalized project root so monorepo projects warm independently, and cache failures fall back to a clean install. - Jobs support dependencies through `needs`, with real-time logs, cache timing telemetry, and cancel/rerun controls. ### Forge Deploy preview Forge Deploy hosts authorized public or private repository applications at `*.sites.smol.ai` with immutable assets, an HTTP Worker, Forge-managed durable state, bounded realtime, exact-SHA previews, automatic or manual publication, and pointer-based rollback. Applications describe their provider-neutral shape in a typed, statically evaluated `forgeBuild.ts`. Monorepos may also add a static root `.forge/config.json` to map project roots, shared defaults, and reviewed Cloudflare binding proposals. It never replaces the per-project `forgeBuild.ts` or grants builds DNS authority; inspect/import and provider **Apply** remain separate administrator actions. See the [repository configuration contract](docs/deploy/repository-config.md). Cloudflare remains the runtime provider; Forge adds the Git-native release control plane. It binds application code and assets to one exact commit, creates an immutable preview before production, blocks automatic activation when the branch has advanced, records the complete release receipt, and rolls back by moving a production pointer instead of rebuilding an old checkout. Start with the public [Forge Deploy preview guide](docs/deploy/forge-deploy-preview.md), the live [`forgeBuild.ts` v1 specification](https://forge.smol.ai/spec/deploy/v1), or the three [conformance fixtures](examples/forge-deploy/README.md). Large public binaries can remain standard Git LFS pointers backed by Forge Assets: OpenNext plans their exact provider identity from verified digests and streams only provider-requested bytes instead of buffering a release in memory. ### AI transcripts and safety - Transcript upload accepts raw JSONL from Codex, Claude Code, Cowork, Cursor, Factory Droid, and other agent formats that can be parsed server-side. Devin and OpenCode can be captured through CLI best-effort SQLite extraction when their local stores are available. - Commit linkage works through explicit `commit_sha` uploads and `AI-Session` git trailers. - Secret masking supports built-in credential masks plus global and repo-specific patterns before transcript display. - Content safety includes blocklists and reports for repository content. ### Gists - Signed-in users can create public or unlisted snippets containing up to ten root-level text files. - Public Gists appear in discovery. Unlisted Gists are link-readable, not private, and do not appear in public discovery. - Owners can edit or delete their Gists; dedicated personal access token scopes support API clients. - Per-file, per-Gist, per-account, and write-rate limits bound D1 storage use. See the [Gists guide](docs/guides/gists.md) for limits and API behavior. ### Schema coverage The database spans users, repositories (including visibility, fork ancestry, and SPDX license metadata), refs, issues, comments, labels, pull requests, collaborators, organizations, teams, team members, stars, Gists and Gist files, workflow configs, workflow runs, jobs, steps, logs, artifacts, secrets, transcripts, secret masks, webhooks, content blocklists, and content reports. ## Project Structure ```text forge/ +-- config/ | +-- forge-components.json # Component ownership and dependency graph | +-- forge-releases/ # Per-component runtime contracts +-- packages/ | +-- edge/ # Public routing, web assets, blog, RSS, social cards | +-- content/ # Public content and blog metadata | +-- web/ # React product UI | +-- identity/ # Identity Worker entrypoint and provider contract | +-- repository/ # Git/repository Worker entrypoint and provider contract | +-- content-control/ # Gists, skins, and product social metadata | +-- deploy-control/ # Deploy control Worker entrypoint and provider contract | +-- notifications/ # Notification Worker entrypoint and provider contract | +-- api/ # Internal implementation kernel during domain source extraction | | +-- src/components/ # Component composition modules | | +-- src/git/ # Git protocol and storage internals | | +-- src/features/ # Domain implementations | +-- runner/ # Trusted CI/build orchestration and provider adapter | +-- deploy-runtime/ # Hosted application runtime | +-- sites-worker/ # Hosted-site edge routing | +-- wiki-worker/ # Wiki execution boundary | +-- ai-router/ # Model routing + owned AI usage D1 ledger | +-- slack-agent/ # Slack agent boundary | +-- cli/ # `sf` / `smolforge` CLI | +-- shared/ # Versioned cross-component contracts +-- migrations/ # Canonical control schema plus generated domain schemas +-- scripts/ # Validation, impact, contracts, and recovery tooling +-- docs/ # Public guides plus excluded operational material +-- examples/forge-deploy/ # Deploy conformance fixtures ``` ## Development Timeline Forge began as a single-Worker project built in one extended session. This is the origin story; the current component architecture is described above. 1. **Scaffolding** — Used ralph-claude-code to bootstrap the monorepo structure, set up Hono, and generate initial route stubs. 2. **Git Protocol (the hard part)** — Implemented the Smart HTTP protocol from scratch: packfile parsing with a custom deflate stream scanner, delta resolution (OFS_DELTA + REF_DELTA), pkt-line encoding, and object graph walking. This required reading RFC 1951 (DEFLATE), RFC 1950 (zlib), and the git pack format spec. 3. **API Routes** — Built out all CRUD endpoints: repos, issues (with auto-increment numbering and comments), pull requests, branches, commits, file browsing, and collaborators. 4. **Frontend** — 18-page React SPA with TailwindCSS: login/register, dashboard, repo view with file browser, issue tracker, PR management, branch list, commit history, and settings. 5. **Deployment** — Deployed the Worker to Cloudflare. Hit several issues: - Wrangler CLI's `/memberships` endpoint failed with API tokens → switched to Cloudflare REST API for D1/R2 creation - `git clone` returned "bad line length character: PACK" → fixed sideband-64k encoding, then simplified to raw pack data without sideband - `git clone` returned "bad object" → the `want` line parser wasn't stripping capabilities from the SHA - JWT_SECRET was a `[vars]` binding conflicting with `secret` → removed from vars, set via `wrangler secret put` - Frontend served on same domain via `[assets]` in wrangler.toml, eliminating CORS issues 6. **Testing** — 119 unit tests (pkt-line, objects, delta, packfile) + 20 API E2E tests + 32 privacy E2E tests, all passing. 7. **Privacy Fixes** — Found and fixed private repo info leaking via issues endpoint (returned data instead of 404 for unauthorized access). 8. **Organization/Team System** — Designed and implemented org-based auth: organizations, teams, team-repo mappings, role hierarchy (owner > admin > member), and permission levels (read < write < admin). See the [organizations and teams design](docs/engineering/organizations-design.md) for the full design. ## AI Agent Integration (`llms.txt`) SmolForge ships a comprehensive `llms.txt` file designed for AI coding agents (Claude Code, Codex, etc.) to consume programmatically. It includes: - **Complete API reference** — every endpoint with auth requirements, request/response formats, and example payloads - **Transcript hook automation** — CLI-managed hooks for Claude Code, Codex, Cursor, Factory Droid, Devin, OpenCode, Aider, Continue, generic JSONL agents, and Git hook backstops - **Backfill & incremental workflows** — bulk-upload all existing transcripts or hook into your commit flow for continuous uploads The recommended setup is CLI-managed: ```bash sf hooks install alice/demo --agent all --git post-commit,pre-push sf hooks status alice/demo ``` This creates a repo-local upload script, installs native lifecycle hooks where available, and installs Git hook backstops for Codex and other agents. Codex is supported through the Git hook path and `~/.codex/sessions` JSONL discovery. Git has no native `post-push` hook, so SmolForge uses `pre-push` for the final missed-upload check. See [`llms.txt`](llms.txt) for the full guide. ## CLI SmolForge also ships an installable CLI package, `@smolai/forge`, with two equivalent binaries: `smolforge` and `sf`. For new JavaScript and TypeScript repositories, Forge strongly recommends pnpm as the default package manager. Bun is the high-speed alternative when project compatibility allows. npm and Yarn repositories remain fully supported; Forge's UI and CLI provide advisory guidance and never silently migrate a project. ```bash # npm npm install -g @smolai/forge # Homebrew, once the tap is published brew install smol-ai/tap/smolforge # Local formula from this repo brew install --HEAD ./Formula/smolforge.rb ``` Basic usage: ```bash smolforge login --username alice sf repo create demo --description "Edge-hosted git repo" git remote add origin $(sf clone-url alice/demo) git push origin main # Install transcript capture before agents start writing code sf hooks install alice/demo --agent all --git post-commit,pre-push sf hooks status alice/demo # Upload Codex, Claude Code, Cowork, Cursor, Factory Droid, or other JSONL agent transcripts sf transcript upload alice/demo --file ~/.codex/sessions/2026/03/29/session.jsonl # Work with Actions and the live machine-readable API guide sf actions runs alice/demo --limit 5 sf llms --save llms.txt ``` The CLI defaults to `https://forge.smol.ai` and can be pointed at another deployment with `--base-url` or `SMOLFORGE_BASE_URL`. ## Releasing Forge Normal production releases use `forge-release/v2`. Feature work submits an immutable pull-request SHA; affected CI produces signed evidence; the protected merge queue is the only authority allowed to advance `main`; and one resumable controller promotes the affected components under a Durable Object fencing token. ```bash SHA="$(git rev-parse HEAD)" sf release enqueue "$SHA" \ --ref codex/my-feature \ --repo swyx/forge \ --pr 123 \ --yes \ --json sf release status fmq_request_id --repo swyx/forge --json sf release watch fmq_request_id --repo swyx/forge --json ``` Enqueue records durable canonical state; it does not activate a provider inline. The controller owns managed Workers and connected edge activation, persists phase history and receipts in D1, and rejects stale provider mutations. Component input digests and successful activation receipts determine impact, so unchanged Workers need not share the edge source SHA. The canonical observation commands are `sf release status/watch`. Direct local deployment is audited break-glass recovery, not a normal release path. See the [release state machine](docs/deploy/release-state-machine-v2.md) and [production runbook](docs/operations/production-deployment.md). Queue validation reuses signed exact-candidate evidence and joins an in-flight run for the same production base instead of launching duplicate CI. Managed Workers activate in dependency-safe parallel waves. Each Worker owns its health binding list beside its runtime entrypoint, so adding a component does not edit one global binding registry or select every existing Worker. ## Related Documentation - [Forge engineering blog](https://forge.smol.ai/blog) — Architecture, reliability, Git, CI, and deployment field notes - [Blog RSS feed](https://forge.smol.ai/blog/feed.xml) — Subscribe to new Forge engineering posts - [llms.txt](llms.txt) — Machine-readable API reference and AI agent integration guide - [Documentation](docs/) — User guides, product contracts, engineering reference, design, operations, and project history - [Organizations and teams](docs/guides/organizations.md) — Quick curl examples for the org/team API - [Production deployment](docs/operations/production-deployment.md) — Release, recovery, rollback, and smoke checks - [Character skins](docs/character-skins.md) — Animation families, variants, sprite strips, manifests, and validation ## Running Locally This repository predates Forge's pnpm-first recommendation and intentionally retains its committed npm lockfile until a measured migration is performed. ```bash # Install dependencies npm install # Start the API (with D1/R2 local emulation) cd packages/api npx wrangler dev # In another terminal, start the frontend cd packages/web npm run dev ``` ## Running Tests ```bash npm run typecheck npm test # Local-only unit and integration tests npm run test:migrations # Clean schema plus data-preserving 0015 -> current-head upgrade npm run test:smoke # Real Worker: push, clone, pull, issue, PR, and merge ``` Remote E2E tests are opt-in, require `E2E_BASE_URL` to name a disposable staging deployment, and refuse known production hosts: ```bash E2E_BASE_URL=https://staging.example.test npm run test:e2e:remote:api E2E_BASE_URL=https://staging.example.test npm run test:e2e:remote:web ``` ## Deploying Production changes enter the protected merge queue with `sf release enqueue`. After signed affected CI passes, the queue advances `main` and hands the exact merge SHA to the fenced, resumable controller. A release is complete only when its canonical status is terminal, required components and bindings are healthy, and the public verification receipt is recorded. A Git push or successful build alone is not deployment evidence. See [production deployment](docs/operations/production-deployment.md) for break-glass recovery, rollback, and smoke checks. ## What's Next - Pull request conversations, reviews, approvals, and requested changes - Real merge commits plus squash and rebase strategies - Additional isolated runner images and package-manager cache strategies - Fork-network browsing and optional upstream synchronization ## Archive The `archive/ralph/` directory contains files from the [ralph-claude-code](https://github.com/frankbria/ralph-claude-code) autonomous development loop that was used during initial scaffolding. These are preserved for historical interest — they show the prompts, fix plans, and session logs from the automated build process.