One-stage build: algebraic docs missed the preset quality margin
The agent built a small application in a single stage. With gpt-6-sol, Algebraic docs scored below our control (No docs - code only); with gpt-6-luna it scored above! However, neither difference reached the threshold set before the runs, so the prediction is not supported. Notably, luna on xhigh showed improvements on every measure on its run.
Results at a glance
| Run | Arm | Quality score | Stages compared | Tokens | Time | Python CC |
|---|---|---|---|---|---|---|
| E009 · gpt-6-sol · medium | No docs | 0.590 (best in run) | 1 of 1 | 1.20M (best in run) | 0:07:40 | 6.25 |
| Prose docs | 0.580 (−0.010) | 1.26M (+5%) | 0:07:37 (−1%) (best in run) | 5.92 (best in run) | ||
| Algebraic docs | 0.520 (−0.070) | 1.59M (+32%) | 0:07:51 (+2%) | 7.21 | ||
| E010 · gpt-6-luna · xhigh | No docs | 0.520 | 1 of 1 | 9.28M | 0:30:42 | 5.65 |
| Prose docs | 0.580 (+0.060) | 12.04M (+30%) | 0:40:09 (+31%) | 5.48 | ||
| Algebraic docs | 0.590 (+0.070) (best in run) | 8.63M (−7%) (best in run) | 0:30:18 (−1%) (best in run) | 5.00 (best in run) |
Each score is from the run’s one scored stage. Tokens and time are each run’s builder totals over every stage it worked. Brackets compare with No docs. Highlighted values mark the highest quality score and lowest recorded tokens, time, and Python CC within each run; ties share the highlight. This does not mean the success rule passed. Python CC means the recorded mean cyclomatic complexity of the Python code’s functions, classes, and methods at the final stage. It includes vendored code where present.
What changed in this condition
The agent builds the application from an empty repository, from one imperative prompt: it says what to build and how. Each run uses a different builder model, to test the result at different model sizes.
Control
No docs
No design method and no starting design. Before coding, it writes its own method, then its designs.
Comparison
Prose docs
A design method in prose. No starting charters.
Hypothesis arm
Algebraic docs
A design method in algebra (capability contracts, Hoare triples, laws and proofs). No starting charters.
Runs: 2 (E009, E010), one per builder model.
- E009: gpt-6-sol · medium
- E010: gpt-6-luna · xhigh
Agent harness: codex-cli. Judge: claude-fable-5-1-high.
Stage evidence
Not supported. Algebraic docs needed a quality gain of at least +0.1 over No docs in each run.
| Arm | Quality score | Quality Δ vs No docs |
|---|---|---|
E009 · gpt-6-sol · medium | ||
| No docs | 0.590 | — |
| Prose docs | 0.580 | −0.010 |
| Algebraic docs | 0.520 | −0.070 |
E010 · gpt-6-luna · xhigh | ||
| No docs | 0.520 | — |
| Prose docs | 0.580 | +0.060 |
| Algebraic docs | 0.590 | +0.070 |
Quality Δ is arm score minus No docs score in the same run. A positive value favors the named arm; a negative value favors No docs.
Noise. No two runs here repeat the same setup. In this group’s repeated runs, the same arm’s score at the same stage moved by a median of 0.03 and at most 0.11.
Details
What was measured
The judge scored each arm’s code after the build stage on a 0–1 quality scale. For the verdict, quality Δ is Algebraic docs score minus No docs score in the same run. The target is +0.1 or greater in every scored stage of both runs. Totals shows measured scores, tokens, and time; only the Algebraic docs versus No docs quality difference decides this result.
Runs
E009 — gpt-6-sol · medium
Run ID: run-20260923T172943Z-c25ba500
Started: 2026-09-23
Harness status: completed
Every arm finished every stage.
E010 — gpt-6-luna · xhigh
Run ID: run-20260923T174224Z-7bf8e1fb
Started: 2026-09-23
Harness status: completed
Every arm finished every stage.
Limits
- Each builder model ran once, so neither model’s own run-to-run noise is known.
- The judge here was Claude Fable 5.1. Later conditions used Claude Opus 5.5, so their scores do not compare with these.
Source data
View condition E009-E010 in G001.json. The run IDs above identify the counted runs. See Data and methods for the export method and limits.