Evidence autoresearch

One honest page to judge a single question: does scaffold-selection actually improve safety fitness? Methods, results, and power, read from the git-tracked ledgers (mutations.jsonl + baseline_archive.jsonl) — measured values only, no fabricated numbers. 0 promotions is a trust-increasing result, not a failure: a loop that promotes nothing on null evidence is behaving correctly. Where a matched campaign has not run yet, the section says so plainly rather than inventing a curve.

1 · Methods

The experimental design, honestly. The only valid evidence of cross-generation improvement is the frozen held-out ruler; everything else (the co-evolving selection pool, a single arm's drift, judge noise) is a confound the design isolates.

held-out ruler (E2)

VERSION-FROZEN bench (held_out_bench_id), an older-runs set DISJOINT from the selection pool. held_out_fitness is the SAME 0-1 compute_fitness (HIGHER-is-better) scored on it EVERY cycle. Because the bench never mutates, this curve IS comparable across generations.

selection pool (B2)

the PINNED co-evolving pool pool-68dc6f0c9745 the loop selects on. It co-evolves, so the intrinsic fitness measured on it is NOT cross-generation evidence — only the held-out ruler is.

3 control arms (E3)

gate = selection (promote gate) · random = random-accept control · never = no-mutation floor. The cross-arm comparison on the SAME fixed ruler isolates selection from drift + judge noise: if gate does not beat random + never on the held-out curve, the improvement is not from selection.

replicate + ci-excludes-0 (E4)

per-mutation replicate M (repeated audits of the same cycle) decomposes provider jitter (within) from seed heterogeneity (between). A gain is CLAIMED only when its confidence interval excludes 0 (gain_ci_excludes_zero); otherwise the honest verdict is "no evidence yet".

reproducibility pins (E5)

each cycle records prompt_hash · applied_diff_hash · sampling_params · rng_seed so a third party can reconstruct WHAT WAS SENT (auditability — no backend-determinism claim).

epoch partition (A)

every promoted baseline is hashed into a content-addressed epoch (be-NNN) keyed on the production+measurement spec; baselines from different epochs were produced under different logic and are never averaged into one comparison.

Control arms

2 · Results

Read from the recorded ledgers. The per-cycle held-out curve PER ARM (split by promote_policy), the 3-arm comparison on the fixed ruler, promotion count per arm, and the ci-excludes-0 verdict. Pre-E1 mixed-scale rows (fitness_before > 1.0) are excluded from aggregates. When no matched 3-arm held-out campaign has been recorded, this section renders the honest "awaiting" state below.

3-arm comparison · fixed ruler

Mean held-out fitness (the fixed ruler) per arm. Selection (gate) is only evidenced if it beats BOTH controls on this ruler. Arm-tagged promotions: 0 — 0 promotions is a trust-increasing result: the loop correctly promoted nothing on null evidence.

selection (promote gate) · gate · 17 generations

random-accept control · random · 5 generations

no-mutation floor · never · 10 generations

Gain verdict · ci excludes 0 · 32 recorded

The explicit "ci excludes 0" evidence statement on the fitness gain. gain significant only when the CI lies entirely above 0; otherwise the honest null: no evidence yet.

3 · Live campaign

The running 3-arm campaign, drilled down per cycle. The gen-0 K-mean noise band fixes the floor every arm is judged against; the promote-gate margin rule says why a cycle rejects; the per-cycle table surfaces every value the loop emits — selection vs held-out fitness, their divergence (the winner's-curse signal), the verdict (promote / reject / SKIP), the seeds scored, and the deep-linked Petri eval. The campaign is mid-run, so cells read "in progress" honestly where a value is not yet recorded.

gen-0 baseline noise band · K-mean re-measure

K=5 held-out noise band: 0.8185 ± 0.0055 stderr. An arm beats noise only when its held-out clears mean + stderr (> 0.8240). held-out fitness is 0-1, HIGHER-is-better.

promote-gate margin rule

The gate promotes only when the fitness gain clears its own margin (SoT core/self_improving/train.py::_should_promote). The margin is NOT persisted per-cycle, so the RULE + the recorded baseline stderr are shown rather than a fabricated number.

margin

max(1.0·√(σ_prior²+σ_current²), 0.005 floor, 0.05 if baseline N=1 critical)

fitness scale

0-1 compute_fitness, HIGHER-is-better

gen-0 baseline fitness_stderr

0.0000

gate decision so far

0/16 measured gate cycles have a positive fitness_delta (best -0.0014); promoted 0. A reject when no gain clears the margin is correct, not a tuning failure

per-cycle campaign drill-down · verdict / seeds / Petri eval / divergence

One row per cycle, faceted by arm. verdict + SKIP are the per-cycle latest SoT (campaign-progress.log) — distinct from the promoted champion SoT (baseline_archive.jsonl, the 3-arm table above). divergence = (selection − gen-0 selection) − (held-out − gen-0 held-out): the gap between how much the cycle moved the selection proxy vs the frozen ruler. A positive divergence is the winner's-curse signal (selection rose more than the held-out ruler). selection + held-out are 0-1 fitness (HIGHER-is-better) but on DIFFERENT scales, so they are anchored to their own gen-0 mean before subtracting; the held-out Δ-vs-prior arrow is green when it rises.

selection (promote gate) · gate · promote 0 · reject 16 · SKIP 2

random-accept control · random · promote 0 · reject 5 · SKIP 5

no-mutation floor · never · promote 0 · reject 13 · SKIP 2

4 · Power

How many samples are needed to detect a target effect δ at 80% power, given the observed fitness-noise σ (recorded E4 decomposition). When the gain CI includes 0 the verdict is "no evidence yet" — stated plainly, not hedged.

target effect δ

0.0200 (fitness scale, HIGHER-is-better)

significance α

0.05

target power

80%

observed σ (recorded)

0.0132

required N_seed per arm

7

To detect δ=0.0200 fitness at 80% power (α=0.05), observed σ=0.0132 → need N_seed≥7 × M_replicate≥1.