Milestone 02

THE SECOND PUBLIC PROOF POINT

Selvage learns
to read the codebase.

Milestone 01 proved Selvage could run the engineering process on a greenfield task. Now we put it on a real, multi-service production-style repository and gave the orchestrator a structural map of the code. The map didn't just help — it caught a cross-service bug that a scripted hand-orchestration baseline shipped, and reached the same merged, reviewed result from a single command for about 26% less than that baseline cost to finish.

Cross-service bug
Caught every un-mapped run missed it
Blocked → shipped
$3.82 → $1.38 DB isolation cleared the block
Orientation search
52 → 16 calls before first edit
vs hand-orchestration
~26% less to the same merged result

A single controlled engineering experiment on one real repository — directional, not a benchmark. Cost deltas are from single runs per condition; the behavioral shifts (searches replaced by structured queries, a bug prevented, a block cleared) are the durable findings. Full honest limits below.

WHAT SHIPPED

Four improvements, each measured on the way in.

Every change below landed because a run showed it mattered — not on intuition. Same issue, same models, same frozen evaluator throughout.

52→16 A map instead of a search party

Injected orientation cut exploration before the first edit from 52 tool calls to 16 — and eliminated the sub-agent the un-mapped run spawned just to find where to work.

Blocked → merged Isolation beats recovery

A per-attempt throwaway database turned a run that thrashed to a dead stop ($3.82 spent, nothing shipped) into a clean first-attempt merge ($1.38).

404 → fixed It caught the bug we all missed

Teaching the map to see mirrored routes made the implementer wire the API gateway — the exact line every earlier run, and a human reviewer, let slip.

8 queries Ask the map, don't grep

Exposed as live tools, the index answered the model's questions during the task — it queried the graph 8 times instead of grepping. Same 6/6 quality; on that run, ~28% cheaper than the briefing-only Selvage config.

THE SETUP

One issue. One baseline. One variable at a time.

Every run implemented the same GitHub issue — persist director notifications for overdue assessments — on the same pinned commit of a real repository: four Go services behind an API gateway, a TypeScript frontend, Postgres and object storage under test. Same implementer model, same independent reviewer model (fresh session, cross-model). Everything was graded by a frozen evaluator written before any run and by the project's own build and tests against a container database.

Why this issue: it is deliberately reuse-heavy. Done well, it extends an existing module and its audit path rather than reinventing them — exactly the task where knowing "what already exists and what it connects to" should pay off.

FINDING 1 · ORIENTATION

Without a map, it searches the whole repo to begin.

We measured the work of finding where to make the change — tool calls before the first edit. The map replaces the search party.

Exploration tool calls before the first edit Without the map: 52 tool calls to orient, including glob, grep, and a dedicated search sub-agent. With the map: 16 tool calls and no search sub-agent. TOOL CALLS BEFORE FIRST EDIT NO MAP glob + grep + sub-agent 52 CALLS TO ORIENT 52 WITH MAP 0 glob · 0 grep · 0 sub-agent 16 16 FIRST PASS COST · $1.55 → $0.99
The un-mapped run spawned a dedicated sub-agent whose whole job was to find where to implement. The mapped run went straight to the right files.

FINDING 2 · ISOLATION

The biggest cost wasn't the model. It was a shared test database.

Three of four cycles in an early run were wasted on a shared database leaking rows between tests. We gave each verification attempt its own throwaway database, then ran it head-to-head.

Shared versus isolated verification database
Verification databaseSharedIsolated per attempt
Implementation attempts41
OutcomeBlocked, never mergedMerged, approved
Spend$3.82 (wasted)$1.38

The shared-database run exhausted its revision limit fighting contamination and shipped nothing. The isolated run merged on the first attempt.

Honest scope: the contamination is a property of that repo's test suite, not a Selvage defect. Our fix is a generic, database-agnostic hook — Selvage guarantees a unique id per attempt; the project declares how to build its own clean database. It's opt-in: a suite that already isolates itself needs nothing.

FINDING 3 · THE BUG

Every un-mapped run shipped a 404.
The map fixed it.

The new endpoint lived in one service, but the API gateway mirrors every one of that service's routes as a string-literal allowlist — a runtime convention no import graph can see. Every early run added the endpoint to the service and forgot the gateway. One reviewer caught it; another approved the same omission.

Route mirroring: without and with the map Without the map the client request reaches the gateway, which has no notifications route registered, and returns 404 — the service handler is never reached. With the map's mirror warning, the gateway route is added and the request reaches the service handler. WITHOUT THE MAP Client GET /director/notifications API gateway route NOT registered 404 Service handler exists… …but is never reached. WITH THE MAP Client GET /director/notifications API gateway route mirrored ✓ Service handler notifications served Map flags: “/director/* lives in 2 files — update both.”
The map learned to read HTTP route literals and warn when a route family spans more than one file. With that hint, the implementer wired the gateway — the line every prior run missed. This is the failure class a code map is supposed to prevent.

FINDING 4 · ON DEMAND

Then we let it query the map
whenever it wanted.

A one-time briefing helps at the start; the model still grepped for specifics mid-task. So we exposed the map as live, read-only tools it can call during the work. It queried the graph 8 times — and stopped grepping.

Cost with one-shot briefing versus on-demand map tools Both runs had the briefing, isolation, and route hint, wired the gateway, and scored 6 of 6. Briefing only cost 4.00 dollars; adding on-demand map tools cost 2.87 dollars — about 28 percent less. TOTAL COST · USD · BOTH 6/6, APPROVED, GATEWAY WIRED BRIEFING ONLY 0 map queries · mid-task greps $4.00 + ON-DEMAND MAP 8 map queries · 1 glob · 1 grep $2.87 ≈ 28% LESS · SAME QUALITY
Given the map as a callable tool, the implementer asked it — what provides this, who depends on it, who mirrors this route — instead of grepping, and delivered the same quality for about 28% less on this run.

THE REAL TEST · VS HAND-ORCHESTRATION

One command, or the steps
strung together by hand.

The comparison that matters isn't Selvage vs Selvage — it's Selvage vs doing the orchestration yourself. So we ran the same issue, same baseline, as a scripted stand-in for the hand-orchestrated path: the implementer, the build-and-test step, and a fresh independent reviewer invoked as separate, disconnected steps, with no autonomous glue between them — an automated simulation of that workflow, not a live human at the keyboard. We took both runs all the way to an approved, merged result. Here is what each cost to get there.

Scripted hand-orchestration baseline versus autonomous Selvage run, both finished Both runs were taken all the way to an approved, merged result scoring 6 of 6. The scripted hand-orchestration baseline cost 3.90 dollars because it missed the api-gateway route and needed a second review cycle to catch and fix it. The autonomous Selvage run cost 2.87 dollars and needed only one review cycle, because its code map caught the cross-service gap on the first pass. Selvage was about 26 percent cheaper to reach the same endpoint. COST TO REACH THE SAME MERGED RESULT · USD · BOTH 6/6 HAND-ORCHESTRATED 2 review cycles $3.90 SELVAGE 1 review cycle $2.87 ≈ 26% LESS TO THE SAME ENDPOINT
Both runs finished at the same place — approved, merged, 6/6. Selvage got there for about 26% less because its code map caught the cross-service gap on the first pass; the hand-orchestrated path missed it, then spent a full second review cycle catching and fixing it.
The honest takeaway: to reach the same merged, reviewed result on this task, Selvage cost ~26% less ($2.87 vs $3.90) — not because the model was cheaper per token, but because the map avoided a second review round. Caveats: the baseline is a scripted simulation of step-by-step orchestration (no live human time counted), and this is n=1 — one issue, one repo.

HONEST LIMITS

What this is, and what it isn't.

  • 01
    One issue, one repository.

    Single runs per condition. The direction is consistent and the behavioral shifts are unambiguous, but the cost deltas need a multi-issue repeat before we quote them as headline numbers.

  • 02
    Recovery cycles vary independently of the map.

    Autonomous verify/recover loops add attempts; comparing raw totals across arms with different attempt counts is misleading. We isolated the map's effect per attempt.

  • 03
    The map is structural, not semantic.

    It answers "does this exist, what does it export, who depends on it, who mirrors this route." It is not a search engine over meaning.

  • 04
    Reviewer nondeterminism is real.

    The same omission was caught by one reviewer and approved by another. Deeper, change-scaled review is its own line of work.

  • 05
    The cost claim is narrow — read it precisely.

    "~26% less" is finished-vs-finished on one issue in one repo, and the baseline is a scripted simulation of step-by-step orchestration — it counts model cost only, not the human time a real operator would spend. The saving came from avoiding a second review cycle on this task, not from cheaper tokens. It is a directional result, not a benchmark; a multi-issue repeat is needed before it's a headline number.

THE TAKEAWAY Give the orchestrator a map of the code — as orientation, as a cross-file guardrail, and as a tool it can query on demand — and it works like an engineer who knows the repo. To reach the same merged, reviewed result on this task, Selvage cost about 26% less than a scripted hand-orchestration baseline — because its map caught the cross-service bug first pass and skipped a second review cycle. The model does the coding. Selvage hands it the map — and runs the process.
Run your first task Revisit Milestone 01 View on GitHub