test-time skill optimization · ARC-AGI-3 · prior-work synthesis · 2026-07-30
This repository already holds four prior-system dissection pages at implementation depth. This page does not re-derive them. It fills only the missing synthesis layer — reindexing those systems against our four measured axes (quality, improvement, judging, retrieval) and joining them to the numbers measured today. Every claim deep-links to the existing page that already carries it.
Every cell on this page carries one of the five badges below. A claim the badge does not license cannot enter the cell, and a cell with no basis in this repository is marked not recorded here rather than left blank. No cell is filled from recollection of the literature.
code-in-repo the source is inside this repository · verbatim-quote a verbatim quote of an out-of-repo source, preserved by a dissection page here · self-reported a figure the team reported themselves, unverified · prose-only only a sentence or two of prose here, no implementation · not-recorded-here nothing recorded in this repository
| system | what this repo contains | source code | nature of the figures | badge licensed |
|---|---|---|---|---|
| Schema (Impossible Research) |
Dissection page codenative/references/schema.html (477 lines). Verbatim copies of the published schema-level description are preserved as NODE_DESC strings at :367–371 and as a data-tip at :288. | Code and prompts are absent here and not public. The only public artifact is the HF dataset schema-harness/arc-agi-3-schema-traces. | 98.98% and similar are self-reported, on the public set (25 games) only, from an ensemble including a Fable 5 fallback rerun. | verbatim-quote self-reported |
| baseline1 (astroseger) |
Dissection page codenative/references/baseline1.html (681 lines). Function signatures, exception messages and prompt sections are excerpted verbatim from the source. | A clone exists at /home/v-seungplee/arc-3-agents-baseline1, but it is outside this repository. | This page makes no score claim for it. | verbatim-quote |
| SkillOpt (arXiv:2605.23904) |
Dissection page codenative/references/skillopt.html (652 lines). Signatures and docstrings for the trainer, gate and optimizer are excerpted verbatim. | A clone exists at /home/v-seungplee/SkillOpt, but it is outside this repository. | Benchmark figures (52 cells and so on) are self-reported in the paper. | verbatim-quote self-reported |
| symbolica (agentica) |
Dissection page codenative/references/symbolica.html (422 lines). Our engine is a fork of this codebase — prior work and our own lineage at once. | Inside this repository: engine/agents/templates/agentica/ (prompts.py, scope/roles.py, skills/skill.py, skills/skill_judge.py, sleep/, wake/). | No score claim. | code-in-repo |
| DreamCoder (Ellis) |
One sentence of prose: a reviewer verdict in the experiment ledger at docs/reports/autoresearch-log.md:1180. No implementation cell. | not recorded here | not recorded here | prose-only |
This table is the ceiling for the whole page. A system absent from it appears in none of the sections below.
The four things a skill loop must do — make good rules (quality), repair what it made (improvement),
choose what to believe (judging), and retrieve the chosen rule at the moment it is needed — are
computed directly from the recorded runs by docs/reports/scripts/four_axes.py.
Measured 2026-07-30. Both runs are n=1.
| axis | metric | bwsvfull (ft09, n=1) | ls20a (ls20, n=1) |
|---|---|---|---|
| ① quality | realized F1 of the library's best rule / duplicate code groups | best_f1 0.813 · 4 duplicate groups (max 6) | best_f1 0.832 · 3 duplicate groups (max 19) |
| ② improvement | records revised past version 1 | 0 of 244 | 0 of 124 (control noE3full: 0 of 346, n=1) |
| ③ judging | corr(top_credence, realized F1) / rounds where the top-ranked rule was the best available | −0.305 · 0 of 59 | +0.249 · 3 of 39 |
| ④ retrieval | best vs USED F1 / share of commits made with the top mechanic silent | 0.813 vs 0.267 · silent 67.4% | 0.832 vs 0.328 · silent 62.5% |
bwsvfull's 6/6 finish (59 rounds, 227 actions, RHAE 19.7) cannot be credited to the skill loop. That run won with the expert briefing sections absent for 36 consecutive rounds (CONSENSUS in 0 of 59 rounds; WORLD MODEL and WIN-CONDITION gone after round 23) and with simulate called only 3 times. Winning with three of the four axes not working is itself the finding. The control sits physically beside it: noE3full, with the gate deleted but the ledger unfixed, reached 4 levels in 60 rounds and burned 123 actions on level 4 (human baseline 16) — n=1. Seeds 1 and 2 have never been run, so the 6/6 is unreplicated.
Rows are the systems from §01, columns the axes from §02. Each cell states how that system handles that axis and what it predicts about our measurement, with its badge. Cells with no basis in this repository read not recorded here.
| system | ① quality — best_f1 0.813 / 0.832 | ② improvement — 0 of 244 / 0 of 124 | ③ judging — corr −0.305 / +0.249 | ④ retrieval — USED 0.267 / 0.328 |
|---|---|---|---|---|
| Schema | verbatim-quote The unit of quality is not an individual rule but a single step() program, and the state representation is invented inside it, so a counterexample can indict the representation itself. A per-rule F1 distribution does not structurally exist — our best_f1 statistic is a question Schema never asks. | verbatim-quote “fixing the existing program, not stacking a new record beside it” (schema.html:367). Improvement is guaranteed by identity — the sibling-accumulation failure mode is structurally removed. We lack that device and measured 0 of 244. | verbatim-quote “A single mismatch pinpoints the bug.” (schema.html:368). run_backtest is binary, whole-history certification — there is no rank to become miscalibrated. We have a rank and measured −0.305. | verbatim-quote With one program there is nothing to retrieve; search happens only inside the certified program, and commit_actions is the sole action channel. A silent commit is structurally impossible. We measured 67.4% silent. |
| baseline1 | verbatim-quote Quality is judged binary by pixel-level whole-history replay (np.array_equal, frame against frame). A rule scoring a partial 0.813 is simply a failure here — pass, or VerificationMismatchError. | verbatim-quote Codex edits the same four files in place, and from level 2 upward a 7-shot simplification pgroup (“hard refactoring”) is forced every cycle. Improvement history accumulates on file identity. Ours: 0 of 244. | verbatim-quote “This script — not a human — decides whether the model is right.” (baseline1.html#snode-verify). There is no place for an LLM's self-assessed rank to enter the judgment. For us the self-assessed credence is the rank, and we measured −0.305. | verbatim-quote The model is the working directory, so it is always fully loaded, and plan_executor commits a prediction before every client call (“no real action without a prediction”, enforced in code). A silent commit is structurally impossible. Ours: silent 67.4%. |
| SkillOpt | verbatim-quote The quality signal is a single hard/soft rollout score, global to the whole document. There is no local notion of quality such as per-rule realized F1 — our axis ① is a question SkillOpt never poses. | verbatim-quote It applies bounded edits (add/delete/replace) to one skill document and rolls back to the previous state when the gate rejects. Improvement is always the next version of the same document. Ours: 0 of 244. | verbatim-quote “Analogous to validation-based early stopping and model selection in neural network training” (the evaluate_gate docstring). Acceptance is a strict inequality on a held-out measured score — the generator's self-confidence never enters the judgment. Ours: −0.305 · top = best in 0 of 59 rounds. | verbatim-quote Deployment attaches the full text of best_skill.md as-is (“zero inference-time model calls at deployment” README, verbatim) — there is no retrieval stage at all. Ours: best 0.813 vs USED 0.267. |
| symbolica | code-in-repo An EVIDENCE PROTOCOL is enforced on mining: “>=10-word verbatim quotes from memory.details or trace_digest lines” + coordinate-free (scope/roles.py). Ungrounded mining is designed to be penalised as a grounding failure. In our ls20a, hypotheses about an ACTION5 the game does not have appeared in 19 of 40 briefings — that protocol is absent from our mining path. | code-in-repo Knowledge updates on the same record: the posterior (confirm_n, falsify_n) moves only through confirm()/falsify(), and when falsification dominates the record is quarantined rather than deleted. We forked this lineage and our refine path still stacks siblings (§04-1). Ours: 0 of 244. | code-in-repo combined = advantage × grounding, and “a skill commits ONLY on a confident verdict (combined >= commit_conf)” (skill_judge.py) — fail-closed guards the moment of commit only. Nothing checks credence against realized F1 afterwards; post-hoc miscalibration is structurally open, and our −0.305 is that open door, measured. | code-in-repo Retrieval is treated as an explicit design problem: retrieval_labels (T3), posterior_by_game (T7), and the is_surviving() survival predicate. Our measurement of that same surface is best 0.832 vs USED 0.328 (ls20a). |
| DreamCoder | not recorded here | prose-only “grows in count, not expressive power” (autoresearch-log.md:1180, reviewer verdict of 2026-07-23). Ours: 0 of 244 · duplicate code groups 4 (max 6) / 3 (max 19). | not recorded here | not recorded here |
The upper loop is the six stages of Schema's published description; the lower is the seven stages of our code. The four red cuts sit exactly where the measurements in §02–§03 land. The single green bar on the right is the one point of honest convergence.
Four mechanism claims. Each stands on a verbatim quote of the published description on their side and on our own measurement on ours. The form is fixed: this mechanism structurally removes this failure mode; we do not have it, and we measured that failure mode.
If there is exactly one thing to repair and edits land on it, the failure mode “no revision is ever produced” is impossible by definition.
_with_code() at ttso/sleep/orchestrator.py:1360
returns a new Record instead of calling set_code() on the existing one, so every accepted refinement is stored
as a sibling rather than a version 2. The only set_code call sites are :1267 (goal_reask) and
:1335 (when-gate repair), while the refine path builds its candidate through _with_code at
:1497. The phrase preserved on Schema's dissection page is precisely the name of our anti-pattern.A whole-history replay that returns only pass or fail has no “confidence ranking” layer. With no ranking, there is no failure mode in which the ranking diverges from measurement.
If knowledge lives in one program, the problem “retrieve the right rule at the right moment” does not arise. There is no way for the top knowledge to fall silent at the moment of action.
Certification is used as a licence to act: plans built inside the certified program are drawn from a zero-real-cost search and consumed in batches, with the per-click prediction check acting as the safety pin.
“We kept the half of Schema's idea that prevents damage and lost the half that makes certification worth earning.”presentation/experiments.html §07 (line 682) — a sentence this repository had already written
Schema's code and prompts are absent from this repository and not public. The four claims above stand only on the schema-level description they published (preserved verbatim by schema.html here). The 98.98% is self-reported, covers the public set (25 games) only, and comes from an ensemble that reran sub-80 games with Fable 5. This page makes no line-level implementation comparison and does not claim their mechanisms caused their score. What can be said is exactly the four cells above: mechanism X structurally removes failure mode Y; we do not have X, and we measured Y.
In each of the five pairs below, the left side is a sentence already recorded in the experiment ledger
(docs/reports/autoresearch-log.md) on 2026-07-23 and 07-26; the right side is the four_axes.py
measurement of 2026-07-30. The prediction precedes the measurement — this is not post-hoc fitting.
ls20a(n=1): value-agreement 0.975 (estimator: 1 − Σwrong / Σ(pred∧obs)), best_f1 0.832 — a closed world predicted almost perfectly — and 40 rounds, 40 actions, level 0 uncleared. The trigger stayed silent.
0 plan-kind records — every run, both games. The diagnostics piled up (RESOURCE_LIMIT 22 · CONSENSUS_BLOCKED 11, across bwsvfull's 39 sleeps). None of it ever crossed into action.
The gate-ON arms commit 1.00 actions/round (n=1 each) · bwsvfull USED f1 0.267 vs best 0.813 · silent-commit 67.4%. Certified truth still goes unconsumed.
Records past version 1: 0 of 244 (bwsvfull) / 0 of 124 (ls20a) · duplicate code groups 4 (max 6) / 3 (max 19). The library grew only in count.
bwsvfull's sleep triggers are “wincond” in 39 of 39 cases and “surprise” in 0. The signal went through the same door every time, and nothing in the record shows it changing the direction of action.
Every number in these rows is carried over unchanged from the sections above (each run n=1). The full run ledger is in this week's deck (index.html).
| component | nearest prior | its prediction here | our number |
|---|---|---|---|
| mine | symbolica EVIDENCE PROTOCOL code-in-repo | Only mining grounded in a verbatim quote of ≥10 words passes — an ungrounded hypothesis is a grounding failure | best_f1 0.813/0.832 — but in ls20a a hypothesis about a nonexistent ACTION5 rode 19/40 briefings, 11/40 actor reasonings and 14 attempted submits (all availability-guarded; none reached the engine). Its source is the base prompt's interface list, not the bundle |
| refine | Schema theorize verbatim-quote | With in-place repair, version 2 exists by identity | 0 of 244 · proximate cause: _with_code creating siblings at orchestrator.py:1360 |
| credence | SkillOpt evaluate_gate verbatim-quote | Judgment by held-out measured score only — self-assessed rank excluded | corr(top_credence, realized F1) = −0.305 · top = best in 0 of 59 |
| retrieval | Schema's single program verbatim-quote | With knowledge in one place, retrieval failure is undefinable | best 0.813 vs USED 0.267 · silent 67.4% |
| brief | symbolica's unconditional handover (“Always pass GAME_REFERENCE … to every subagent.”) code-in-repo | Passing knowledge on is a contract, not a choice | CONSENSUS 0/59 · OTHER SKILLS gone after round 6 · WM/WIN-CONDITION gone after round 23 (truncated by wake.py cap_chars=8000; fixed 07-29, unvalidated) |
| simulate | Schema run_bfs (“real cost is zero”) verbatim-quote | Free exploration should be consumed hard — RHAE falls with the square of actions | bwsvfull 3 times — all in one round (r53), with 0 across the other 58 · C3 in 41 rounds (simulate does not increment game.actions = the only free exploration) |
| plan | baseline1's planner + resimulation self-check verbatim-quote | A certified plan is a licence that can be spent | 0 plan-kind records (every run, both games) · bindings RESOURCE_LIMIT 22 · CONSENSUS_BLOCKED 11 |
| commit | baseline1 plan_executor — predict·execute·compare verbatim-quote | Check the prediction at every click, discard the plan on first mismatch — the honest convergence | The pin at namespace.py:378–394 implements the same discipline (added 2026-07-16; C3 predates it) · bwsvfull batched in 43 of 59 rounds |
| sleep | SkillOpt's nightly cycle — one night = one epoch, budget-bounded verbatim-quote | Sleep should respond to a need signal and have bounded cost | Triggers 39/39 “wincond”, “surprise” 0 · sleep is 81% of LLM calls · non-LLM compute per round 1.5 → 116 minutes (O(n²) in rules×buffer) |
| record | Schema Timeline (“append-only … the only ground truth”) verbatim-quote | The record is what the next certification replays | The jsonl ledger persists — but there is no certify stage that replays it in full; wm_fidelity.jsonl exists only from bwsv onward (noE3full values are marked provisional) |