Prior Work Deep-Dive · grounded in source code

baseline1 (astroseger) — a world-model agent that edits files

Every signature and quote on this page is taken directly from the arc-3-agents-baseline1/secure_baseline1 source. Nothing is filled in by guesswork; where uncertain, the original text is quoted verbatim.

01

The system in one line

A system that uses unsandboxed Codex CLI as a file-editing agent and forces the world model into four persistent files. An external controller (agent.py) pushes situation-appropriate prompts into Codex; Codex never plays the game directly — it keeps editing the world-model files in the working directory, and whether the model is right is decided not by a human but by verify_world_model.py: every recorded attempt is simulated from scratch, and the rendered ASCII frames must match the real observations pixel for pixel to pass. §02 below splits this cycle into five stages: observe → write world_model.py → planner → replay verify → attempt log.

AGENT.md’s self-description (verbatim): “Our agent is based on unsandboxed Codex. … agent.py is the external controller. It prepares the run directory, starts Codex, sends prompts to Codex, runs the client, inspects progress, and decides which prompt to send next.” Isolation is handled by two Docker containers (agent/server) and the client–server boundary, so the agent can never peek at the arc_agi library (the game implementation).

02

The pipeline — a 5-step click stepper

The outer loop is the controller’s prompt state machine; the inner loop is the “predict → execute → compare → repair” cycle Codex performs. The Action-discipline clause of main_prompt.md enforces the inner loop in prose: “predict the settled result of the next action … if mismatch exists, stop and repair the model. Never continue blindly after a mismatch.” Hover a step below for a preview; click to pin its code schema and TTSO counterpart in the right panel.

The critique loop (a standing discipline outside the steps): main_prompt.md demands a “Generalization-critique subagent” fed critique_prompt.md verbatim — run an adversarial review whenever the model grows complex, whenever a special case is tempting, and before ever declaring a level solved (mostly at the STEP 2 / STEP 4 boundary). Even the critic’s output format is fixed: Findings / What Seems Sound / Bottom Line.

03

Key function cards — signatures verbatim from source

world_model_engine(state, action) → (new_state, game_status)

src/agent/workspace_init/world_model_engine.py — the provided stub, in full (8 lines)

from game_status import RUNNING

def world_model_engine(state: dict, action: dict) -> tuple[dict, str]:
    """Placeholder world-model dynamics."""
    return dict(state), RUNNING

The game dynamics within a single attempt. The agent fills this stub with the real rules. state is a dict with a mandatory "level" field; action is a dict of name (+ x,y for ACTION6). The returned status is one of RUNNING | LEVEL_COMPLETED | GAME_OVER.

input → output example

world_model_engine(
    {"level": 1, "avatar": (12, 40), "lives": 3, ...},   # agent-defined internal state
    {"name": "ACTION6", "x": 15, "y": 47},
)
# -> ({"level": 1, "avatar": (15, 47), "lives": 3, ...}, "RUNNING")
# on the goal-reaching step -> (terminal_state, "LEVEL_COMPLETED")

initial_state_reconstruction(level_index, initial_frame) → dict

src/agent/workspace_init/world_model_state_io.py

def initial_state_reconstruction(level_index: int, initial_frame: np.ndarray) -> dict:
    """Placeholder initial-state reconstruction."""
    return {"level": int(level_index)}

The single entry point that reconstructs internal state from a level’s initial observed frame. Exactly one file-loading path is allowed — load_initial_full_frame(level_index) for partially-observed levels (main_prompt, verbatim: “This rule is strict and non-negotiable.”). Any later state is produced only as “initial state + replaying actions through the engine,” never by another function.

input → output example

initial_state_reconstruction(2, frame)   # frame: (64,64) int16, values 0-15
# -> {"level": 2, "walls": {...}, "avatar": (7, 9), "baseline_lives": 3, ...}
#    (for games with death-resets, baseline values for fresh-attempt rebuilds are planted here too)

state_renderer(state) → np.ndarray  /  apply_render_overrides(frame, state, level_index, attempt_index, step_count)

src/agent/workspace_init/world_model_state_io.py

def state_renderer(state: dict) -> np.ndarray:
    """Placeholder renderer."""
    return np.zeros((64, 64), dtype=np.int16)

def apply_render_overrides(frame, state, level_index, attempt_index, step_count) -> np.ndarray:
    """Verification-only escape hatch for unresolved frame-specific mismatches."""
    return frame

Internal state → the expected ASCII frame (64×64 int16). Verification is possible only through this function, so the model’s “acceptance criterion” is renderer agreement. apply_render_overrides is a last-resort patch hook that temporarily papers over unexplained visual detail — and the verifier prints a warning whenever the hook changes a frame: “treat this as temporary and as a clue to a missing puzzle mechanic”.

input → output example

state_renderer({"level": 1, "avatar": (15, 47), ...})
# -> array([[0,0,...],[...]], dtype=int16)  # shape (64,64); must match the real frame exactly to pass

verify_world_model.py — _replay_attempt(…) and VerificationMismatchError

src/agent/workspace_init/verify_world_model.py

class VerificationMismatchError(AssertionError):
    """Expected verification mismatch; distinct from world-model implementation failures."""

# verification core: initial-frame render check -> per-step (status, frame) check
state, game_status, rendered = _predict_step(state, step, attempt_index, index + 1)
if game_status != step["observed_status"]: _raise_status_mismatch(...)
if not np.array_equal(rendered, step["final_frame"]): _raise_frame_mismatch(...)

Replay-verifies every attempt across levels 1..N. If even one frame differs, it saves five kinds of artifacts and throws an exception carrying their paths. The whole verification is bounded by TIMEOUT_SECONDS = 180; exceeding it prints “Your world model is too slow, please consider refactoring it.”

output on failure (exception message format, strings verbatim from source)

VerificationMismatchError:
Model level 3, replay level 2, attempt 1, replay, step 14: rendered frame mismatch.
real ascii: .../client/session/level_2_attempt_1/step_0014_final.txt
real png: .../step_0014_final.png
simulated ascii: .../mismatch_frames/7/simulated_frame.txt (predicted frame as text)
simulated png: .../mismatch_frames/7/simulated_frame.png (predicted frame rendered)
mismatch region png: .../mismatch_frames/7/mismatch_region.png (localized mismatch neighborhoods)
mismatch as magneta png: .../mismatch_frames/7/mismatch_as_magneta.png (full frame with magenta mismatch pixels)

“magneta” reproduces the source’s own typo — file names, function names, and messages all consistently spell it magneta.

save_mismatch_region_png_v1(real_frame, predicted_frame, output_png_path, scale) → Path

src/agent/workspace_init/frame_plot_lib.py — MISMATCH_RADIUS = 7, BROWN_RGB = (139, 69, 19), MAGENTA_RGB = (255, 0, 255)

def save_mismatch_region_png_v1(real_frame, predicted_frame, output_png_path, scale=DEFAULT_PNG_SCALE) -> Path:
    # Shows only neighborhoods around mismatch pixels; everything else is plotted in brown,
    # a color outside the game palette. (source comment, verbatim)

def save_mismatch_as_magneta_png_v2(real_frame, predicted_frame, output_png_path, scale=DEFAULT_PNG_SCALE) -> Path:
    # Paints exact mismatch pixels as magenta on the full real frame.

Two mismatch visualizations. v1 keeps only a radius-7 neighborhood around each mismatch pixel in true color and paints everything else brown — a color outside the game palette — pinning the eye on “where it differs.” v2 stamps just the mismatch pixels magenta on the real frame — useful when mismatches are scattered.

input → output example

save_mismatch_region_png_v1(real, predicted, Path("mismatch_frames/7/mismatch_region.png"))
# -> PosixPath('mismatch_frames/7/mismatch_region.png')
#    (a PNG where only radius-7 around mismatch cells is true color; background brown (139,69,19))

plan_executor.py — the predict-execute-compare loop

src/agent/workspace_init/plan_executor.py — the core main() loop (excerpt)

for action in actions:
    predicted_state, predicted_status = world_model_engine(current_state, action)
    predicted_frame = state_renderer(predicted_state) if predicted_status == RUNNING else None
    subprocess.run(_client_command(action), cwd=CLIENT_DIR, check=True)   # executes on the real game
    ...
    if actual_status != predicted_status:
        raise AssertionError(f"plan_executor.py: status mismatch after {format_action(action)}: ...")
    ...
    if predicted_frame is None or not np.array_equal(predicted_frame, observed_frame):
        _raise_frame_mismatch(...)   # saves artifacts, then AssertionError

The only channel that executes a planned action sequence on the model and the real game simultaneously. Each step’s prediction is committed before the client is ever called, so the “no real action without a prediction” rule is enforced at the code level. RESET is unsupported (explicitly rejected).

input → output example

$ python3 plan_executor.py ACTION1 ACTION1 '{"name": "ACTION6", "x": 15, "y": 47}'
plan_executor.py: running ACTION1
plan_executor.py: running ACTION1
plan_executor.py: running ACTION6 x=15 y=47
plan_executor.py: sequence executed without mismatch      # or: halts at the mismatch + prints artifact paths

read_all_attempts_for_level(level_index, session_dir=None) → list[Attempt]

src/agent/workspace_init/session_tools.py — Attempt / AttemptStep TypedDicts (verbatim)

class Attempt(TypedDict):
    level_index: int;  attempt_index: int;  path: str
    initial_frame: np.ndarray;  initial_metadata: dict[str, Any]
    initial_frame_filename: str;  initial_frame_png_filename: str
    steps: list[AttemptStep];  status: str

class AttemptStep(TypedDict):
    action: dict[str, Any];  final_frame: np.ndarray;  metadata: dict[str, Any]
    final_frame_filename: str;  final_frame_png_filename: str
    intermediate_frame_filenames: list[str];  intermediate_frame_png_filenames: list[str]
    observed_status: str

Reads and structures what the client left in client/session/level_<N>_attempt_<A>/ (per-step metadata JSON + ASCII/PNG frames). ASCII frames parse one character = one hex digit (int(char, 16) → an int16 array). This is the sole data source for the verifier and for state reconstruction.

input → output example

read_all_attempts_for_level(2)
# -> [ {"level_index": 2, "attempt_index": 1, "status": "GAME_OVER",
#      "initial_frame": array(64x64), "steps": [ {"action": {"name": "ACTION1"},
#      "final_frame": array(64x64), "observed_status": "RUNNING", ...}, ...]}, ... ]

planner(state) → list[dict] | None

src/agent/workspace_init/world_model_main_planner.py — the stub, in full

def planner(state: dict) -> list[dict] | None:
    """Placeholder planner."""
    return None

Searches the internal world-model state for an action sequence to level completion; returns None if it cannot. run_main_planner.py must re-simulate the returned plan through the engine and self-verify that it reaches LEVEL_COMPLETED (AssertionError otherwise) — “no planning outside the model” enforced in code here too. Auxiliary planners may take a goal dict, as planner(state, goal=None).

input → output example

planner({"level": 1, "avatar": (12, 40), ...})
# -> [{"name": "ACTION1"}, {"name": "ACTION1"}, {"name": "ACTION6", "x": 15, "y": 47}]
# unreachable -> None
04

The situational prompt state machine (21 prompts/*.txt)

Every iteration, the controller classifies state by looking only at the session directory — it never trusts the LLM’s self-report; the filesystem is the truth. agent.py :: iteration_loop() branches four ways: ① game-over → reset_protocol, ② level changed (except level 1) → new_level_protocol, ③ step count unchanged since last iteration → stuck_protocol, ④ otherwise → normal_continuation_protocol. On top sits a trouble 1/2 escalation keyed to steps accumulated at reset (+100 steps per level → trouble1, +200 → trouble2 — trouble2 tears down even the Codex session via runner.new_session() and restarts from main_prompt).

prompt filewhen the controller sends it
main_prompt.mdat start (with the game’s initial screen printed), on recovery, and on trouble2 (as the opener of the new session)
continuation_string.txtthe tail of nearly every prompt: “keep going without stopping until LEVEL_COMPLETED or GAME_OVER”
continuation_level1.txt / continuation_l2.txtthe continue instruction under normal progress. Level 1 gets the “a simple puzzle where a few probes reveal the answer” version; level 2+ gets “think in world-model terms and refresh the planner”
light_simplification_level1.txton level 1, right before every continuation/reset/stuck — one light simplification review
world_engine_simplification_step1–3.txt
world_model_state_io_simplification_step1–3.txt
world_model_planner.txt
on level 2+, right before every continuation/reset/stuck: a 7-shot pgroup (“hard refactoring”) — 3 engine steps + 3 state_io steps + 1 planner step of compression/generalization orders
on_new_level_v1.txtthe first iteration after the level index changes: orders a full-element inventory of the new level’s first frame (classified known / new / uncertain)
death_prompt.txtsent first on entering reset_protocol via GAME_OVER: “fold why you died into the model; inspecting the animation PNGs is mandatory”
trouble1_prompt.txt / trouble1_prompt_level1.txtwhen reset after burning 100+ steps on the same level since the last trouble1
trouble2_prompt.txt / trouble2_prompt_level1.txtwhen reset after burning 200+ steps — a fresh Codex session is created and rebooted with main_prompt + trouble2 (“you may have tunnel vision”)
stuck_reminder_prompt.txtwhen a whole iteration passes with zero game steps: “if you don’t move, you lose.” If still no progress, stop_condition ends the run
recovery_prompt.txton a --recovery-mode restart: orders a review of model, logs, and game state plus a situation report
reset_prompt.txt, light_simplification.txtthe files exist but are absent from agent.py’s load_prompt call list — apparently unused on the current controller path (honesty notice)

Stop conditions (stop_condition): current-level steps ≥ 1500, no progress even after the stuck protocol, game cleared, or the current level cannot be found.

05

What we adopted / what we did not

✓ adopted

  • localize → mismatch() — the mismatch-artifact idea of localizing the difference between predicted and observed frames and shoving it in the agent’s face. baseline1’s save_mismatch_region_png_v1/_raise_frame_mismatch are the prototype of our engine’s mismatch-localization primitive.
  • critique → sleep critic — we moved critique_prompt.md’s adversarial generalization review (“are you hiding level-specific memorization?”) into the critic of our sleep phase.
  • curriculum → prompt block A — we absorbed main_prompt.md’s clause “levels are a curriculum: later levels extend earlier mechanics, so the level-N model must stay valid on all of 1..N” into our prompt block A.

✗ not adopted

  • Codex file-editing orchestration — the whole structure of keeping the world model as four .py files on disk edited by a CLI agent. Our executable world model lives as miner-authored predict_next skills scored on the game’s own logged transitions (Game.fork stays sealed), so no separate file layer is needed.
  • The Docker isolation scheme — the agent/proxy/server three-container structure (Dockerfile.agent/proxy/server) for running unsandboxed Codex safely. Not applicable to our execution model.