WHAT WE CAN ACTUALLY CLAIM
Same score. Half the wall clock.
Nothing forgotten at the seam.
We re-ran both arms of our Milestone 02 experiment on the same afternoon — the hand-orchestrated baseline and Selvage — against the same frozen issue and the same hidden evaluator. Both reached the same objective score. What separated them was not tokens: it was one command instead of five interventions, one review cycle instead of two, and a cross-service route that the orchestrated arm has now wired first-pass in every run where its code map was switched on.
- Time to a reviewed result
- 6m51s vs 14m10s across four legs
- Cross-service route
- 3 of 3 with the route-mirror signal · 0 of 5 finished runs without it
- Review cycles to approval
- 1 vs 2 the second existed to fix a miss
- Frozen evaluator
- 6/6 both objective quality is parity
One issue, one repository, single runs per arm — the same limits Milestone 02 carried. The route-mirror finding is the exception: it now holds across nine runs. Cost moved too, and we discuss it below, but it is the claim we trust least and the reason is instructive.
ACCURACY · THE FINDING THAT REPEATS
Nine runs later, the seam
tells the same story.
The feature lives in one service. The API gateway mirrors that service's routes as a string-literal allowlist — a runtime convention no import graph can see. Add the endpoint, forget the gateway, and every test passes while the endpoint returns 404 to the outside world.
| Run | Code map | Route wired first pass |
|---|---|---|
| Run 1 · manual | none | No |
| Run 1 · Selvage | orientation only | No |
| Run 2 · shared DB | orientation only | Never finished — excluded |
| Run 2 · isolated DB | orientation only | No |
| Run 5 · manual | none | No — reviewer caught it |
| Run 6 · manual | none | No — reviewer caught it |
| Run 3 · Selvage | + route mirror | Yes |
| Run 4 · Selvage | + route mirror | Yes |
| Run 6 · Selvage | + route mirror | Yes |
Five finished runs without the mirror signal, five misses. Three runs with it, three catches. A sixth run without it never finished — it died on database contamination before review — so we exclude it rather than bank it as a miss. A human reviewer caught the omission twice and a different reviewer approved it once — which is the point: this is a class of defect that survives careful review, and a structural signal catches it before review begins.
SPEED · WHAT AN OPERATOR ACTUALLY WAITS FOR
Six minutes unattended,
or fourteen with your hands on it.
The orchestrated arm was submitted with one command and produced a reviewed, verified, approved result 6 minutes 51 seconds later. The hand-orchestrated arm took 14 minutes 10 seconds across four model legs, and needed a human between every one of them.
| Measure | Hand-orchestrated | Selvage |
|---|---|---|
| Model time | 12m09s | — |
| Wall clock to reviewed result | 14m10s | 6m51s |
| Human interventions | 5 | 1 command |
| Legs the operator sequences | 4 | 0 |
| Review cycles | 2 | 1 |
2.07× faster on wall clock. The gap is not model throughput — it is the elimination of the gaps between steps, where a human writes the review prompt, runs the build, reads the verdict, and decides what happens next. Ours were seconds because they were scripted; a real operator's are minutes, and they are the reason a two-cycle task eats an afternoon.
COST · WHY WE DON'T LEAD WITH IT
The cheapest number to move
is the one you don't control.
Milestone 02 reported about 26% less than the baseline. On this re-run the gap is 52.8%. We are deliberately not putting that on the front page, and the reason is the whole point of this note.
Milestone 02's baseline was measured against claude --model sonnet and --model opus. Those are aliases: they resolve to whatever is current. Comparing a fresh Selvage run against a four-day-old dollar figure would have credited us for someone else's release. So we re-ran both arms, same afternoon, same aliases.
| Arm | Sep 8 | Sep 12 | Change |
|---|---|---|---|
| Hand-orchestrated | $3.9043 | $3.4117 | −12.6% |
| Selvage | $2.8706 | $1.6099 | −43.9% |
| Gap vs the baseline | −26.5% | −52.8% | +26.3 pts |
Newer models made the manual path 12.6% cheaper on their own. Had Selvage received only that same model-driven improvement it would cost $2.51 today; it costs $1.61. Model progress alone would have left the gap where it was, at about 26.5% — so the 26.3 points it widened by are the part that sits on our side of the line. That is a real result, and it is still a single run of a single issue whose dollar value the next price change rewrites. Speed and a missed route are properties of the process. Cost is a property of the invoice.
WHAT CHANGED ON OUR SIDE
Better retrieval replaced
the questions entirely.
Milestone 02's last improvement exposed the code index as live tools the model could query mid-task; it used them eight times and stopped grepping. Since then we replaced bulk summary injection with bounded structural-plus-lexical retrieval, which scored 3.4× the recall of the ranking it replaced (0.733 against 0.217) on ten authored queries in one repository — a small, self-labeled benchmark, not an independent one.
| Behavior | Hand-orchestrated | Selvage, Sep 8 | Selvage, Sep 12 |
|---|---|---|---|
| Glob + grep across the run | throughout | 1 + 1 | 0 + 0 |
| Index queries mid-task | n/a | 8 | 0 |
| Orientation handed over up front | none | 1351 B / 11 symbols | 1910 B / 17 symbols |
Zero searches and zero index queries. The tools were connected and allowlisted; the model never needed them, because the orientation it received up front already answered what it used to ask. On-demand querying was not the destination — it was how we learned which questions the model asks, which is what let us put the answers in front of it instead.
HONEST LIMITS
What this is, and what it isn't.
- 01One issue, one repository, one run per arm.
Only the route-mirror finding has a real sample (nine runs). The speed and cost figures are single runs, exactly as in Milestone 02, and the multi-issue repeat we promised remains the bar for a headline number.
- 02It is the same issue as Milestone 02.
Deliberate — holding the task fixed is what isolates model drift — but it also means the system has now been aimed at this task repeatedly. A genuinely new task is the harder test.
- 03The speed comparison flatters the baseline.
Its five human gaps were scripted and took seconds. A real operator reading a verdict and writing the next prompt takes minutes. We report the fast version because we can measure it, not because it is representative.
- 04Objective quality was parity, not a win.
Both arms scored 6/6 on the frozen evaluator. The accuracy claim is specifically about what reaches the first review — a route the tests could not see — not about better code.
- 05Reviewer nondeterminism is unresolved.
The manual arm's reviewer requested changes; Selvage's approved. Part of that is a real defect in one arm and not the other, and part is variance we cannot yet bound.
- 06Selvage's arm finished on evidence, not on a merge commit.
It passed its checks, passed review, and scores 6/6 in a clean checkout — but a defect in our local auto-merge left the final merge to a human. This run is how we found it. It is filed, not hidden.
- 07The rig cost money to rediscover.
The CLI's permission semantics had changed since September 8, and four aborted legs — $3.76 — went into finding the posture that lets a scripted baseline run its own tests. That spend belongs to neither arm and is excluded from every figure above. A measurement you cannot cheaply repeat is one you will quietly stop repeating.