Why a protocol at all
Vendor benchmarks tell me how a model does on somebody else's code. I want to know how it does on mine, with my rules, my test command, and my pack. So I run it as a measurement, not as a vibe.
The protocol is small enough that I actually run it.
The protocol
Pick three closed tickets that have clear tests and a known pre fix commit. Real tickets, already solved by a human, so a correct answer exists to compare against.
For each candidate model, create a fresh worktree at the pre fix commit, apply the red test file, and run the same lane pack through the runner. Same pack for every model. The pack is the control.
Score every run with an Opus 4.8 validator on six things: tests green yes or no, diff scope respected yes or no, no unrelated edits yes or no, findings counted by severity, wall time, and a one to five quality score with a one line reason.
Promote a model to a row only after three of three green runs with zero P1 findings. Record every run, including the ugly ones.
The routing table today
Dated 2026-09-09.
| Task class | Where it goes | Standing |
|---|---|---|
| Mechanical: renames, index regen, doc moves, formatting | OpenCode glm-5.3-flash | Draft |
| Research, code reading, summaries | Claude Sonnet | Draft |
| Scoped build, red tests, small blast radius | OpenCode glm-5.3, then Codex gpt-5.6-sol, then Codex gpt-5.6-terra | Calibrated for glm-5.3, sol and terra |
| Hard build: editors, meetings, concurrency, crypto, shared contracts, unclear scope | Claude Opus 4.8 | Calibrated for Opus 4.8, Codex gpt-6-astra in calibration |
| Validation of every unit | Claude Opus 4.8 validator | Calibrated for the Opus validator |
| Prod touching: migrations, secrets, deploys, payments | Never dispatched, gates on me | Standing rule |
Codex gpt-5.6-sol is my first Codex choice. Codex gpt-5.6-terra is second and faster, and I watch its output for em dashes because it typed one into a doc comment after an explicit instruction not to.
The hold list
Three models are on hold, and none of them get a lane until they earn one back.
- kimi-k2.7-code on OpenCode: a P1. It deleted two localization calls and hardcoded English.
- gpt-5.6-luna on Codex: a P1, the same localization deletion, plus it stripped a spec doc comment.
- gpt-5.6-luna on OpenCode: a P2. Fastest lane I have measured, which is exactly why it stays on the list until it passes properly.
Being fast is not a reason to promote. It is a reason to run the protocol again.
The honest gap
One round was never scored at all: Codex terra hit the ChatGPT usage limit on 2026-09-08 and all three lanes returned the quota error. It was scored the next day, on a pack that named the localization rule, and passed. I keep the unscored day in the record because a calibration log with no failures in it is a marketing page.
Set it up
- Pick three closed tickets in your own repo with real tests and a known pre fix commit.
- Write one lane pack per ticket. Use the same pack for every model you measure.
- For each model, create a fresh worktree at the pre fix commit and apply the red test.
- Dispatch through the runner so the tests run outside the agent's sandbox.
- Score each run with an Opus 4.8 validator on the six items above. Write the score down.
- Promote only on three of three green with zero P1 findings. Otherwise put the model on the hold list with the reason.
- Re-run the protocol when you change the pack. A pack change moved two of my models from a P1 to a clean run, so the old scores were about the pack as much as the model.
What you can do now
You can prove, with numbers from your own repo, which cheap model is safe for which class of work.
Course done
That is the setup. What comes next is not a module, it is a running log: the field notes, updated whenever something breaks or surprises me.
Talk to Claude instead of typing to it.
Ducktate turns what you say into clean text, on your Mac, and pastes it anywhere.
Get early access