Are Harnesses Worth the Cost?
Are Harnesses Worth the Cost?
Naresh Jain had Codex build the same greenfield Java API service repeatedly from a detailed brief. Half the runs also gave the agent machine-readable interface contracts, examples, dependency mocks and executable checks during development. Every finished service faced the same separate 178-check acceptance suite, which was not available to the building agent. On ten attempts per condition, the stronger GPT-6 Sol passed all ten with or without the harness; the cheaper GPT-6 Luna passed three unaided and ten with it. The failures clustered around timeout recovery, polling links and exact wire contracts, rather than basic compilation.
The trade-off reverses by model. The Sol harness runs took nearly three times as long and cost about seven times as much in estimated model usage as its unaided runs, without a measured gain in passing outputs. Harnessed Luna took longer and cost more than unaided Luna, but its estimated model cost per run was lower than unaided Sol while both passed all ten attempts in this sample. Jain suggests giving stronger models an independent check only at the end and weaker models continuous feedback; he did not run the final-only comparison.
This is one unusually complete brief, ten attempts per cell and a computational oracle derived from supplied requirements—not a maintained service with an independent requester. The dollar figures are API-equivalent estimates, not bills; they exclude writing and maintaining the contracts, triaging failed runs, human review, deployment and later repairs. Guides and sensors changed together, so the experiment cannot isolate which part helped Luna. It tests a verification cadence, not the total cost of delivered software.