# E015-E017: Five-stage builds: algebraic docs missed the quality margin and used more tokens

Condition: Generation · fresh codebase · 5 imperative stages · Result: not supported · 2026-09-23

The agent grew one application over five stages of sequential imperative prompts, each in a new container that reused the previous container's environment. Algebraic docs moved up in some stages and down in others, always by less than the threshold, against No docs and against Prose docs. It used the most tokens of the three arms in both runs.

## Results at a glance

| Run | Arm | Average score | Stages compared | Tokens | Time | Python CC |
|---|---|---:|---:|---:|---:|---:|
| E015 · gpt-6-astra · xhigh | No docs | **0.370** | 4 of 5 | **20.47M** | **2:11:12** | 4.11 |
|  | Prose docs | 0.362 (−0.008) |  | 22.82M (+12%) | 2:18:53 (+6%) | **4.09** |
|  | **Algebraic docs** | 0.363 (−0.007) |  | 35.88M (+75%) | 2:57:17 (+35%) | 4.26 |

| Run | Arm | Average score | Stages compared | Tokens | Time | Python CC |
|---|---|---:|---:|---:|---:|---:|
| E016 · gpt-6-sol · medium | No docs | 0.312 | 4 of 5 | 15.00M | 1:00:28 | **6.35** |
|  | Prose docs | **0.325 (+0.013)** |  | **14.25M (−5%)** | **0:50:33 (−16%)** | 7.32 |
|  | **Algebraic docs** | 0.310 (−0.003) |  | 20.49M (+37%) | 1:06:58 (+11%) | 7.14 |


Scores average the stages every arm of the run scored: the stages compared. Tokens and time are each run's builder totals over every stage it worked. Brackets compare with No docs. Highlighted values mark the highest quality score and lowest recorded tokens, time, and Python CC within each run; ties share the highlight. This does not mean the success rule passed. Python CC means the recorded mean cyclomatic complexity of the Python code's functions, classes, and methods at the stage immediately before Rust conversion.


## What changed in this condition

The agent grows the application from an empty repository over five stages of imperative prompts. The last stage converts it to Rust. Each run uses a different builder model, to test the result at different model sizes.

- **No docs** (control): No design method and no starting design. Before coding, it writes its own method, then its designs.
- **Prose docs**: A design method in prose, and four starting charters in prose that define the application's core concepts: checks, operations, the ontology and telemetry.
- **Algebraic docs** (hypothesis arm): A design method in algebra (capability contracts, Hoare triples, laws and proofs), and the same four charters in algebra, each with a written proof.


**Runs:** 2 (E015, E016), one per builder model.

- **E015:** gpt-6-astra · xhigh
- **E016:** gpt-6-sol · medium

**Agent harness:** `codex-cli`. **Judge:** `claude-opus-5-5-high`.


## Stage evidence

**Not supported.** Each part of the rule must hold in every run.

Stages:

1. `initial-build`
2. `telemetry-add`
3. `parity-substrate`
4. `process-ontology`
5. `rust-conversion`

**✗ Algebraic docs must score at least 0.1 above No docs in every graded stage, in every run: not met**

| Run | Arm | Met | 1 | 2 | 3 | 4 | 5 |
|---|---|---:|---:|---:|---:|---:|---:|
| E015 · gpt-6-astra · xhigh | **Algebraic docs** | 0 of 4 | +0.05 | +0.02 | −0.05 | −0.05 | both failed |
| E016 · gpt-6-sol · medium | **Algebraic docs** | 0 of 4 | 0.00 | +0.08 | −0.08 | −0.01 | both failed |


Each cell is the arm's score minus the other arm's at that stage. ✓ marks a stage that meets that part of the rule; Met counts them, of the stages with a value. “failed” means that arm's grade failed at that stage, so the stage has no score.


**Noise.** No two runs here repeat the same setup. In this group's repeated runs, the same arm's score at the same stage moved by a median of 0.03 and at most 0.11.


## Details

### The prediction, set before the runs

The test card names the arms as follows: `control` is No docs, `semantic` is Prose docs, `algebraic` is Algebraic docs; `total_tokens` is tokens and `elapsed_s` is time.

> **Prediction.** The algebraic arm's quality score exceeds the semantic arm's by at least 0.1.
>
> **Falsifier.** The algebraic arm's quality score is no higher than the semantic arm's. A positive difference below 0.1 is inconclusive.

**Measured:** Each arm's judged `score` at each stage. Quality Δ is the arm's score minus No docs. A stage meets the quality target at +0.1 or above. Every counted check must pass in every run. The prediction above names Prose docs; the condition result uses the No docs control.


### Runs

#### E015 — gpt-6-astra · xhigh

**Run ID:** `run-20260923T223518Z-020373a4`  
**Started:** 2026-09-23  
**Harness status:** failed

The Algebraic docs arm's grade failed at stage 5 (`rust-conversion`): `cargo test` failed. The No docs arm's grade failed at stage 5 (`rust-conversion`): `cargo test` failed. Every earlier stage was graded.

#### E016 — gpt-6-sol · medium

**Run ID:** `run-20260923T223651Z-b952ede3`  
**Started:** 2026-09-23  
**Harness status:** failed

The Algebraic docs arm's grade failed at stage 5 (`rust-conversion`): `cargo test` failed. The No docs arm's grade failed at stage 5 (`rust-conversion`): `cargo test` failed. Every earlier stage was graded.

The harness marks a run *failed* when any arm's grade fails. That stage has no score for that arm; the run's other stages still count.


### Limits

- The two runs used different builder models, so they are not repeats.
- Both runs lost the final stage's comparison: at least one of the two arms failed its Rust grade.

### Source data

[View condition `E015-E017` in G001.json](https://mikereese.dev/data/experiments/G001.json). The run IDs above identify the counted runs. See [Data and methods](https://mikereese.dev/experiments/g001/#data-and-methods) for the export method and limits.