Pre-registered, isolated experiments on coding agents

Problem

“You have enough ideas to kill this company.” – Jeff Wilke, to Jeff Bezos early on at Amazon. I have a lot of ideas. I need a process to rapidly validate whether they’re worth pursuing - and the best process for that is the scientific method. The experiment harness exists to create a controlled environment where I can rapidly test ideas via pure-signal experiments. Goal is to quickly design an experiment and a scrap environment to test it in, measure the outcome against a control, and look for an outsized measurable impact before deciding to implement it in a production system.

First principles

• Code generation moves at machine speed. So why not take advantage of that? Telemetry on everything. Rapid, disposable repositories. Low-friction command center for humans to rapidly register repos and configure experiments.
• Has to be a pure signal. Experiment arms may only differ in exactly one configuration slot; everything else is held fixed. With experiment creation being low-friction, allows us to do lots of pure-signal runs concurrently rather than trying to test everything at once.
• It has to represent reality as closely as possible. It’s a literal isolated, ephemeral clone of the system. Done via “Test System” and “Test Environment” pairs in Docker containers. Separate arms are isolated as well - they share no network, file or process. The grader sees exactly what the agent wrote and cannot change it.
• Rules are fixed at experiment design stage. The claim, prediction, the falsifier, and the decision rule are fixed at experiment start and do not change. We test within those bounds only.
• Quantitative and Qualitative Measurements. What can be measured quantitatively is done so mechanically; what needs judgment gets an eval - a judge with frozen rubrics for the entire experiment set. Both are necessary for a full picture of the impact.

Approach

Config-driven arms. A manifest names the system and the grading environment by repository and commit, and each arm by its configuration. The harness refuses a manifest whose arms differ in more than one slot.

Isolated, dockerized arms. Each arm runs as its own containers in a private network namespace, each with its own filesystem. The agent writes its workspace read-write. After it stops, the grader reads the same directory through a read-only mount, so it sees exactly what was written and cannot change it. Only images that pass a conformance episode can launch.

Quantitative and qualitative, side by side. The system records its own cost (tokens, time) and, where the algebra carries proofs, how many z3 obligations hold. The grading environment runs the mechanical checks (tests, lint, lines, complexity) and, separately, a judge that scores against frozen rubrics on seven dimensions, with its full report kept. Every value is a row in a SQLite store committed to git, and a local observer draws each run.

Skills I taught myself

• Experiment design for noisy, qualitative systems.
• LLM Evals with weighted grading and specific categorical assignment criteria.
• Container isolation: Paired containers, network egress config, lifecycle management.
• z3, and SMT solvers, to see if the algebraic specifications idea requiring mechanical validation reduced inference.

Result

21 experiments and 27 runs so far, grouped by causal question. No positive result for algebraic specs yet, and the nulls are published: see Experiments and E022.