Behavioral gates, cheap-model routing and production acceptance

Research brief and result, October 14, 2026. Can executable checks make a smaller coding model a better choice per maintained accepted change, or merely move the labor of building and maintaining the oracle out of view? A good answer compares identical requests with a strong-model baseline, counts all rejected attempts and setup/maintenance labor, and follows independent behavioral acceptance, reviewer decisions and later repair. Our cost ledger says a synthetic router can look cheap per pass without seed cost and human review; Belz says green tests can encode an unreal browser. Answer so far: an independent computational gate can recover specified edge cases for a cheaper model, but a model-cost-per-attempt advantage is not an end-to-end price of trusted change. This case does not follow a maintained feature to requester use.

Four subquestions and short source list. (1) Who authored the requirements and held-out oracle? Read Naresh Jain’s October 13 original methods and tables: he supplied a detailed written brief to all, and contracts, examples, a skill and mock-dependent executable checks only to the harnessed agents. A separate 178-check external suite evaluated all outputs after generation and was not visible during building. He founded the company whose tool instantiated the harness; the comparative result is inspectable as an author-run experiment, not third-party validation. (2) Does feedback save enough to cover its own costs? Rechecked the same article’s token, elapsed-time and dollar tables and its explicit omissions; no final-gate-only arm, paid human oracle time or post-merge observation. (3) Can a gate enforce an architectural rule in real work? Huntley’s October 13 first-person account describes a pre-commit diff check rejecting bypasses of his diagnostic module after instructions drifted. His ‘ten out of ten’ has no published denominator or separate acceptance protocol and is not a controlled estimate; SQLFluff’s accepted but deferred review warning illustrates that a review finding needs an owner before release. (4) Does this generalize to high-volume agent engineering? Anthropic’s October 13 original operator account describes semantic approval rules and post-merge/pre-deploy testing in a bespoke source-control stack, but offers no measure of reviewer touch time or independently accepted customer work. Searches for each question surfaced no same-request full-cost trace; stop generic pass-rate searches.

Primary-checked comparison. Ten isolated Codex runs each for GPT-6 Sol and cost-optimized Luna, with and without the same package, built one greenfield Java API service with detailed interface and failure behavior. All 10 unaided Sol outputs passed the held-out 178 checks; so did all 10 harnessed Sol outputs. Adding feedback made Sol runs about 19m53s and $2.679 (estimated API-equivalent) versus 6m51s and $0.406 without; no detectable acceptance gain in this sample. Unaided Luna passed 3/10 at $0.086 and 15m56s per attempt; with the harness it passed 10/10 at $0.273 and 35m per attempt. Harnessed Luna was approximately 33% cheaper per generation run than unaided Sol in this sample, but roughly four times its own unaided token usage and more than double its own elapsed time. Failures clustered at deferred 202/poll links, error contracts, wire headers and configuration seams. Some reported downstream test failures stem from one missing monitor link and should not be counted as separate independent defects.

Economics and disagreement. The source itself cautions that dollar estimates are not billed spend and exclude writing/maintaining contracts, examples and assertions, plus diagnosing/reworking failures. At face value, $0.086 / (3/10) ≈ $0.287 of model-only attempts per passing unaided Luna run if retries are independent and free to inspect—nearly the $0.273 harnessed Luna figure, but this arithmetic is not a measured repeat strategy and ignores latency, triage, iteration and sample uncertainty. Conversely an always-on harness on Sol imposed nearly sevenfold estimated model cost without a passed-check improvement. Jain proposes final verification only for stronger models, but that arm was not run. The guide and sensors varied together, so their separate effects are unknown. Ten outcomes per cell on one unusually specified project cannot establish the long-run pass rate or benefit on partial briefs and brownfield software. The external evaluator was separate from construction but encoded the same supplied requirements, not a new requester's independent judgment; a green suite can still miss the real system Belz encountered. No human reviewer minutes, merge, deploy, customer confirmation or 30-day repair were captured.

Next decisive test. On the same real request, compare (a) strong model with final-only gate, (b) cheap model with continuous gate, (c) cheap model with final-only gate and repair, with the same independently owner-checked contract and unanticipated real-client scenario. Attribute specification/fixture writing and maintenance, all attempts, review touch minutes, machine usage and waiting, then inspect behavior and repair 30 days later. Retain an explicit human disposition when the requester's intended behavior differs from the formal contract. Until then, use this as a result about verification cadence and model choice within a fixed, specified test task, not a claim that the agent reduced engineering labor or that review is obsolete. See review attention and decision→oracle chain.