← index week of 2026‑07‑17 · TTSO · experiment bestiary — 7 run types (2026‑06‑30 → 07‑17) history →

test-time skill optimization · ARC-AGI-3 · ft09 · experiment bestiary

Seven ways this system checks its own claims

TTSO does not trust a run because it happened — every credence update, every sleep, every full-game attempt is gated by one of seven falsification patterns. This page is the catalog: one card per instance, each reading problem → design → measured result → evidence. Every number below is copied from an already-published page or a raw log in this repo — nothing here is invented to fill a gap.

C3 6/6 · C2c 6/60710 double WIN — full ft09, unseeded, fork_dependence=0 both arms (§③ below)
00

Scope & honesty note

window

2026-06-30 → 2026-07-17. Sources read to assemble this catalog:

honesty note

Categories ① and ⑥ were requested with specific named examples (“0705 D2 seeded L4 / --load-bundle”, “ls20 27 runs”) that do not appear verbatim anywhere in the sources above. Rather than invent them, each of those two sections below reports the closest real, sourced analog actually found in these files, flagged explicitly at the top of the section.

Sleep-round rate per run, jsonl-matched: C3 WIN 10/52 (19%), h9 0716 19/22 (86%), h10 0717 14/30 (47%)
sleep-round rate per run (“SLEEP: mined” round count / total rounds, jsonl-matched) — C3 WIN 10/52 (19%) · h9 0716 19/22 (86%) · h10 0717 14/30 (47%). note: these jsonl-matched round counts (n=22 for h9, n=30 for h10) run slightly longer than the final embedded-ledger figures cited in ①–⑦ below (h9 rounds=20/sleep=16, h10 rounds=26/sleep=9) — both are real measurements of the same two runs, taken at different aggregation points (raw jsonl-round match vs. final run ledger). Both are reported verbatim, not reconciled.
① unit tests ② isolated falsification ③ full-game unseeded ④ seeded / preload escape ⑤ matched-arm A/B ⑥ constitution audit ⑦ mining battery
①

Unit / module tests (pytest)

1 item

S0 gate — structural invariants before any live budget is spent

problemGuarantee the harness's structural invariants (single fold() α/β gate, byte-identical accept/αβ/V if judge text is deleted/corrupted, real per-skill EIG, idempotent settle_wincond_round) hold before any live run spends budget.
designpython3.12 -m pytest ttso/tests -q as a hard S0 gate — no live run starts on a red suite (constitution §3 invariant, test-enforced).
resultGreen-suite size grew as fixes landed across the window: 321 green (2026-07-10, C-v4 live wiring, commit 9e5fbf6) → 440 tests collected (2026-07-14, branch ttso-nofork-predict-verify) → 516 green (2026-07-15, tense-separation redesign, commit 3008c0d) → 518 green (2026-07-15, VACUITY 2-line prompt fix, commit c0bac81) → 606 green (2026-07-16, constitution-audit blocker fix, commit 357afa8).
evidence
②

Isolated falsification / replay

2 items

0/21 → 13/21 rescoring — the numpy-boundary fix (commit a524931)

problemLive run c0716h8_thin_s0: 6 of 7 rounds forced into sleep, yet every skill's credence stayed frozen at the park prior (0.25) for all 6 sleeps — no engine evidence was folding in at all.
designRescore the same recorded transition offline, before vs. after a one-line fix, to isolate whether the boundary (not the model) was the cause.
resultRoot cause: the judge's _Predictor passed the board to LLM-authored predict_next code as a list-of-lists; 13 of 21 healthy skills assumed numpy (.shape) and silently raised, scoring 0. Same transition rescored: pre-fix 0/21 skills applied → post-fix 13/21 applied, f1=1.0 for 4 of them. Fix = one line (np.asarray) at the scoring boundary, commit a524931.
evidence

0→4 “WIN” correction — retroactive fork_dependence audit

problemAn early sleep-harness run reported a 0→4 level climb (“WIN”, 2026-06-30); needed to verify whether that climb reflected real skill-mining or a leak.
designRetroactive fork_dependence audit / correction pass on the already-published run page, not a fresh live run.
resultConfirmed leak: the sleep loop had been letting candidate rules query game.fork() (a free, throwaway copy of the real engine) before committing — a fork-bruteforce artifact, not genuine skill inference. The page was corrected in place with a banner rather than deleted; structural fix: Game.fork sealed to raise RuntimeError on call, fork_dependence=0 logged and asserted every turn thereafter.
evidence
③

Full-game unseeded runs (C3 / C2c WIN, h4–h10)

2 items

RHAE (first use below) = level-weighted clear-efficiency score, Σℓ·scoreℓ/Σℓ, scoreℓ=min((baselineℓ/actionsℓ)²×100, 115); higher = fewer actions relative to a human baseline. Full formula + worked example: metrics.html.

0710 double WIN — C3 & C2c, both fork = 0

problemDoes the fork-free (leak-free) predict-verify loop actually clear a full ft09 game (L0–L5) end to end, unseeded, on one frozen model?
designSingle live full-game run, no ground-truth access, fork banned, budget-capped.
resultC3_fullgame_prob_s0: WIN 6/6 levels, 52 rounds, 174 actions, RHAE 40.9, fork_dependence=0, plan_replay_dependence=0, simulate used in 41/52 rounds. C2c_fullgame_prob_s0_r3: WIN 6/6 levels, 74 rounds, 171 actions, RHAE 55.7, fork=0 — a second, independent full-game WIN forming the “double WIN”.
evidence

0716–0717 verification battery — 7 runs (h4, h8–h10 substantive; h2/h3/h5/h6/h7 discarded)

problemAfter the 0710 double WIN, does the same scaffold reproduce, and do the week's structural fixes (memory-starvation caps, numpy-boundary fix, GAME_OVER-as-falsifier, τ patience-2, judge exec-scoring cap) actually reconnect the learn→score→trust→act loop live?
design7-run verification battery, each rerunning the live full-game loop with the week's fixes incrementally wired in; h2/h3/h5/h6/h7 discarded as transport-layer noise (rate-limit 429 / server 504, no substantive rounds).
result0/7 full clears (win=true) across the battery; level-level clears did occur (best: h9 reached L2, with an L0 clear at r5 and a live L0→L1 rule-transfer observation at r6–r7). GAME_OVER occurred 3 times (h9 r9; h10 r9 and r19). h4 (S5 reproduction) stalled at L3 for rounds 14–30 with a when-gate TypeError repeating 18 times (fail-closed). Final ledger: h4 rounds=30/actions=67/levels=3/sleep=21; h9 rounds=20/actions=77/levels=2/sleep=16; h10 rounds=26/actions=75/levels=1/sleep=9.
evidence
④

Seeded / preload escape

2 items · closest real analogs
honesty note

The exact example named for this category (“0705 D2 seeded L4”, --load-bundle) does not appear verbatim anywhere in the sources read. No CLI flag or run tagged that way is cited on these pages. The two items below are the closest real, sourced analogs found instead.

A — GT-keyed replay rejected, replaced by real-engine predict-verify (pre-0709)

problemPre-2026-07-09: whether letting the sleep loop consult the real engine's hidden ground truth during candidate-rule replay (“GT-keyed replay”) is a legitimate way to pre-validate skills before spending live budget.
designGT-keyed (ground-truth-preloaded) replay of candidate rules, evaluated against the true engine state rather than a fresh observe/act cycle.
resultRejected as unrealistic — a replay that secretly consults GT only produces the illusion of having solved it. Replaced by real-engine observation + LLM inference (U0 observability / U1 descent / U2 uncontrolled-convergence predict-verify, commit 4807091), after which the loop reached L3 for the first time as a genuine (non-leaked) result.
evidence

B — S5 reproduction run, tagged under runs/S5/

problem2026-07-16, S5: whether the same scaffold that produced the 0710 double WIN reproduces when rerun from scratch (escaping back into previously-reached L3 territory) without adding new capability.
designReproduction run tagged and logged under runs/S5/ (the EXPERIMENT_PLAN.md “transfer” stage directory), same thin-actor scaffold as the WIN run, no seed/capability change.
resultDid not reproduce the WIN: stalled at L3 for rounds 14–30 (raw log runs/S5/c0716h4_thin_s0.jsonl), diagnosed post hoc as a memory-starvation bug (briefing truncated a winning claim at [:160]), not a capability gap.
evidence
⑤

Matched-arm A/B (arm on/off, h9 vs h10)

2 items

h9 vs h10 — three-part 0717 revision against the same scaffold

problemh9 exposed two bottlenecks: (a) after a GAME_OVER, the resumed round could not treat the just-fatal goal-claim as falsified (“no-learning gap”); (b) sleep execution cost exploded with store size (15→34 min/round). Does a targeted 3-part revision fix both without new capability?
designPaired live runs on the same scaffold: h9 = pre-revision (13 commits: morning constitution-restoration stack + afternoon starvation/mandate/gate/boundary fixes) vs. h10 = h9 build + 3 additional 2026-07-17 revisions — GAME_OVER promoted to a false_clear-grade falsifier; surprise-trigger τ requires 2 consecutive misses instead of 1; judge execution-scoring capped to top-12-by-credence + 6 new.
resulth10 vs h9: sleep firing rate 4/4 → 1/4 (r17 snapshot), minutes/round 16 → 5. h10 demonstrated the falsifier-promotion working live — a GAME_OVER demoted all 6 competing wincond hypotheses at once (β25, β27, commit abe2a2c). Neither arm reached a level clear=win; wincond-authoring quality remained the open, unresolved variable in both arms. Final ledger: h9 rounds=20/sleep=16/levels=2; h10 rounds=26/sleep=9/levels=1.
evidence

Formal protocol — S3 paired E0/E1, S4 component ablations

problemFormal protocol for any component change: does a fix (E1) beat frozen pre-fix code (E0) on the same game/model/budget without regressing the hard fork=0 invariant?
designS3 (paired seeds 0,1,2, E0 vs E1, same everything else) and S4 (one component ablated at a time against the S3-winning config) as defined in the run book.
resultProtocol-level (not yet a completed battery in these files): metrics fixed as RHAE, L0 rounds-to-clear, fork_dependence==0 (hard invariant both arms), reasoning-rubric score, citation rate, and the VoI-probe falsifier_follow_rate (its judge-eval-text variant flagged as not yet landed).
evidence
⑥

Audit (constitution 1:1)

1 item

5-agent 1:1 clause audit — 22:47 KST, 2026-07-16

problemAfter a day of rapid live-driven patching (13 commits on 2026-07-16 alone across two stacks), verify the implementation still matches ttso/constitution.md — the project's sole normative source — clause by clause, rather than trusting that each individual fix composed correctly.
design5-agent workflow performs a 1:1 audit of the constitution's clauses against the running code (22:47 KST, 2026-07-16).
result23 of 29 clauses OK; 5 confirmed GAPs (0 refuted); 1 blocker found — fold_judge_scores had a 4th credit channel that let negative narrative-DL judgments leak into wincond credence, violating §2 (“static fit folds in neither direction”). Patched red-green same day (commit 357afa8, 606 tests green). The blocker only affected the LLM-scoring path, so the code-scoring h9 run was unaffected by it; the remaining 4 minor gaps were deferred to the next battery.
evidence
⑦

Mining battery

1 item · closest real analog
honesty note

The exact example named for this category (“ls20, 27 runs”) does not appear in the sources read. ls20 pages in this repo belong to an earlier, separate diagnostic phase (agent misreading a fuel bar as a code grid), preserved as a control/reference — not part of the ft09 mining story. The item below is the closest real, sourced mining-battery data: per-run sleep/mining counts across the same 0716–0717 verification battery used in ③/⑤ above.

Per-run sleep/mining counts, 0716–0717 battery

problemWhen the actor is stuck (no level gain for K rounds, or a residual-magnitude trigger), does the LLM-free execution-gated sleep loop (mine ≤3 candidate skills → fork·execute·judge → commit iff engine level-completion rises) actually produce and retain usable skills across a run, and at what mining cost?
designPer-run sleep/mining counts logged across the same battery: how many of each run's rounds triggered a mining cycle, and how long each cycle took.
resultSleep counts across the battery (final ledger): h4 = 21 sleeps / 30 rounds (all stop_reason=plateau); h8 = 6 sleeps / 7 rounds (all frozen at prior 0.25, the bug found in ② above); h9 = 16 sleeps / 20 rounds, with one sleep log noting “mined 11: 0 accepted, 6 parked, 2 confirmed, 0 dropped, 26 observations folded, 15 competing mechanics (entropy 2.59)”; h10 = 9 sleeps / 26 rounds after the 0717 revisions cut the firing rate to 1/4. Miner ledger itself was capped at 400 entries (commit 91a0797) after unbounded growth was diagnosed as one of the five 2026-07-16 starvation surfaces.
evidence