Prior Work Deep-Dive · grounded in source code
Every signature and quote on this page is taken directly from the
arc-3-agents-baseline1/secure_baseline1 source. Nothing is filled in by guesswork;
where uncertain, the original text is quoted verbatim.
A system that uses unsandboxed Codex CLI as a file-editing agent and forces the world model into
four persistent files. An external controller (agent.py) pushes
situation-appropriate prompts into Codex; Codex never plays the game directly — it keeps editing the
world-model files in the working directory, and whether the model is right is decided not by a human but by
verify_world_model.py: every recorded attempt is simulated from scratch, and the rendered
ASCII frames must match the real observations pixel for pixel to pass. §02 below splits this cycle
into five stages: observe → write world_model.py → planner → replay verify → attempt log.
AGENT.md’s self-description (verbatim): “Our agent is based on unsandboxed Codex. …
agent.py is the external controller. It prepares the run directory, starts Codex, sends prompts to Codex,
runs the client, inspects progress, and decides which prompt to send next.”
Isolation is handled by two Docker containers (agent/server) and the client–server boundary, so the agent
can never peek at the arc_agi library (the game implementation).
The outer loop is the controller’s prompt state machine; the inner loop is the “predict → execute → compare → repair” cycle Codex performs. The Action-discipline clause of main_prompt.md enforces the inner loop in prose: “predict the settled result of the next action … if mismatch exists, stop and repair the model. Never continue blindly after a mismatch.” Hover a step below for a preview; click to pin its code schema and TTSO counterpart in the right panel.
The critique loop (a standing discipline outside the steps): main_prompt.md demands a
“Generalization-critique subagent” fed critique_prompt.md verbatim — run an
adversarial review whenever the model grows complex, whenever a special case is tempting, and before ever declaring
a level solved (mostly at the STEP 2 / STEP 4 boundary). Even the critic’s output format is fixed:
Findings / What Seems Sound / Bottom Line.
src/agent/workspace_init/world_model_engine.py — the provided stub, in full (8 lines)
from game_status import RUNNING def world_model_engine(state: dict, action: dict) -> tuple[dict, str]: """Placeholder world-model dynamics.""" return dict(state), RUNNING
The game dynamics within a single attempt. The agent fills this stub with the real rules.
state is a dict with a mandatory "level" field;
action is a dict of name (+ x,y for ACTION6).
The returned status is one of RUNNING | LEVEL_COMPLETED | GAME_OVER.
input → output example
world_model_engine(
{"level": 1, "avatar": (12, 40), "lives": 3, ...}, # agent-defined internal state
{"name": "ACTION6", "x": 15, "y": 47},
)
# -> ({"level": 1, "avatar": (15, 47), "lives": 3, ...}, "RUNNING")
# on the goal-reaching step -> (terminal_state, "LEVEL_COMPLETED")
src/agent/workspace_init/world_model_state_io.py
def initial_state_reconstruction(level_index: int, initial_frame: np.ndarray) -> dict: """Placeholder initial-state reconstruction.""" return {"level": int(level_index)}
The single entry point that reconstructs internal state from a level’s initial observed frame.
Exactly one file-loading path is allowed — load_initial_full_frame(level_index) for
partially-observed levels (main_prompt, verbatim: “This rule is strict and non-negotiable.”).
Any later state is produced only as “initial state + replaying actions through the engine,” never by another function.
input → output example
initial_state_reconstruction(2, frame) # frame: (64,64) int16, values 0-15 # -> {"level": 2, "walls": {...}, "avatar": (7, 9), "baseline_lives": 3, ...} # (for games with death-resets, baseline values for fresh-attempt rebuilds are planted here too)
src/agent/workspace_init/world_model_state_io.py
def state_renderer(state: dict) -> np.ndarray: """Placeholder renderer.""" return np.zeros((64, 64), dtype=np.int16) def apply_render_overrides(frame, state, level_index, attempt_index, step_count) -> np.ndarray: """Verification-only escape hatch for unresolved frame-specific mismatches.""" return frame
Internal state → the expected ASCII frame (64×64 int16). Verification is possible only
through this function, so the model’s “acceptance criterion” is renderer agreement.
apply_render_overrides is a last-resort patch hook that temporarily papers over unexplained
visual detail — and the verifier prints a warning whenever the hook changes a frame:
“treat this as temporary and as a clue to a missing puzzle mechanic”.
input → output example
state_renderer({"level": 1, "avatar": (15, 47), ...})
# -> array([[0,0,...],[...]], dtype=int16) # shape (64,64); must match the real frame exactly to pass
src/agent/workspace_init/verify_world_model.py
class VerificationMismatchError(AssertionError): """Expected verification mismatch; distinct from world-model implementation failures.""" # verification core: initial-frame render check -> per-step (status, frame) check state, game_status, rendered = _predict_step(state, step, attempt_index, index + 1) if game_status != step["observed_status"]: _raise_status_mismatch(...) if not np.array_equal(rendered, step["final_frame"]): _raise_frame_mismatch(...)
Replay-verifies every attempt across levels 1..N. If even one frame differs, it saves five kinds
of artifacts and throws an exception carrying their paths. The whole verification is bounded by
TIMEOUT_SECONDS = 180; exceeding it prints
“Your world model is too slow, please consider refactoring it.”
output on failure (exception message format, strings verbatim from source)
VerificationMismatchError: Model level 3, replay level 2, attempt 1, replay, step 14: rendered frame mismatch. real ascii: .../client/session/level_2_attempt_1/step_0014_final.txt real png: .../step_0014_final.png simulated ascii: .../mismatch_frames/7/simulated_frame.txt (predicted frame as text) simulated png: .../mismatch_frames/7/simulated_frame.png (predicted frame rendered) mismatch region png: .../mismatch_frames/7/mismatch_region.png (localized mismatch neighborhoods) mismatch as magneta png: .../mismatch_frames/7/mismatch_as_magneta.png (full frame with magenta mismatch pixels)
“magneta” reproduces the source’s own typo — file names, function names,
and messages all consistently spell it magneta.
src/agent/workspace_init/frame_plot_lib.py — MISMATCH_RADIUS = 7, BROWN_RGB = (139, 69, 19), MAGENTA_RGB = (255, 0, 255)
def save_mismatch_region_png_v1(real_frame, predicted_frame, output_png_path, scale=DEFAULT_PNG_SCALE) -> Path: # Shows only neighborhoods around mismatch pixels; everything else is plotted in brown, # a color outside the game palette. (source comment, verbatim) def save_mismatch_as_magneta_png_v2(real_frame, predicted_frame, output_png_path, scale=DEFAULT_PNG_SCALE) -> Path: # Paints exact mismatch pixels as magenta on the full real frame.
Two mismatch visualizations. v1 keeps only a radius-7 neighborhood around each mismatch pixel in true color and paints everything else brown — a color outside the game palette — pinning the eye on “where it differs.” v2 stamps just the mismatch pixels magenta on the real frame — useful when mismatches are scattered.
input → output example
save_mismatch_region_png_v1(real, predicted, Path("mismatch_frames/7/mismatch_region.png")) # -> PosixPath('mismatch_frames/7/mismatch_region.png') # (a PNG where only radius-7 around mismatch cells is true color; background brown (139,69,19))
src/agent/workspace_init/plan_executor.py — the core main() loop (excerpt)
for action in actions: predicted_state, predicted_status = world_model_engine(current_state, action) predicted_frame = state_renderer(predicted_state) if predicted_status == RUNNING else None subprocess.run(_client_command(action), cwd=CLIENT_DIR, check=True) # executes on the real game ... if actual_status != predicted_status: raise AssertionError(f"plan_executor.py: status mismatch after {format_action(action)}: ...") ... if predicted_frame is None or not np.array_equal(predicted_frame, observed_frame): _raise_frame_mismatch(...) # saves artifacts, then AssertionError
The only channel that executes a planned action sequence on the model and the real game
simultaneously. Each step’s prediction is committed before the client is ever called, so the
“no real action without a prediction” rule is enforced at the code level.
RESET is unsupported (explicitly rejected).
input → output example
$ python3 plan_executor.py ACTION1 ACTION1 '{"name": "ACTION6", "x": 15, "y": 47}'
plan_executor.py: running ACTION1
plan_executor.py: running ACTION1
plan_executor.py: running ACTION6 x=15 y=47
plan_executor.py: sequence executed without mismatch # or: halts at the mismatch + prints artifact paths
src/agent/workspace_init/session_tools.py — Attempt / AttemptStep TypedDicts (verbatim)
class Attempt(TypedDict): level_index: int; attempt_index: int; path: str initial_frame: np.ndarray; initial_metadata: dict[str, Any] initial_frame_filename: str; initial_frame_png_filename: str steps: list[AttemptStep]; status: str class AttemptStep(TypedDict): action: dict[str, Any]; final_frame: np.ndarray; metadata: dict[str, Any] final_frame_filename: str; final_frame_png_filename: str intermediate_frame_filenames: list[str]; intermediate_frame_png_filenames: list[str] observed_status: str
Reads and structures what the client left in
client/session/level_<N>_attempt_<A>/ (per-step metadata JSON + ASCII/PNG frames).
ASCII frames parse one character = one hex digit (int(char, 16) → an int16 array).
This is the sole data source for the verifier and for state reconstruction.
input → output example
read_all_attempts_for_level(2) # -> [ {"level_index": 2, "attempt_index": 1, "status": "GAME_OVER", # "initial_frame": array(64x64), "steps": [ {"action": {"name": "ACTION1"}, # "final_frame": array(64x64), "observed_status": "RUNNING", ...}, ...]}, ... ]
src/agent/workspace_init/world_model_main_planner.py — the stub, in full
def planner(state: dict) -> list[dict] | None: """Placeholder planner.""" return None
Searches the internal world-model state for an action sequence to level completion; returns
None if it cannot. run_main_planner.py must re-simulate the
returned plan through the engine and self-verify that it reaches LEVEL_COMPLETED
(AssertionError otherwise) — “no planning outside the model” enforced in code here too.
Auxiliary planners may take a goal dict, as planner(state, goal=None).
input → output example
planner({"level": 1, "avatar": (12, 40), ...})
# -> [{"name": "ACTION1"}, {"name": "ACTION1"}, {"name": "ACTION6", "x": 15, "y": 47}]
# unreachable -> None
Every iteration, the controller classifies state by looking only at the session directory — it never
trusts the LLM’s self-report; the filesystem is the truth. agent.py :: iteration_loop()
branches four ways: ① game-over → reset_protocol, ② level changed (except level 1) → new_level_protocol,
③ step count unchanged since last iteration → stuck_protocol, ④ otherwise → normal_continuation_protocol.
On top sits a trouble 1/2 escalation keyed to steps accumulated at reset (+100 steps per level → trouble1,
+200 → trouble2 — trouble2 tears down even the Codex session via runner.new_session()
and restarts from main_prompt).
| prompt file | when the controller sends it |
|---|---|
main_prompt.md | at start (with the game’s initial screen printed), on recovery, and on trouble2 (as the opener of the new session) |
continuation_string.txt | the tail of nearly every prompt: “keep going without stopping until LEVEL_COMPLETED or GAME_OVER” |
continuation_level1.txt / continuation_l2.txt | the continue instruction under normal progress. Level 1 gets the “a simple puzzle where a few probes reveal the answer” version; level 2+ gets “think in world-model terms and refresh the planner” |
light_simplification_level1.txt | on level 1, right before every continuation/reset/stuck — one light simplification review |
world_engine_simplification_step1–3.txtworld_model_state_io_simplification_step1–3.txtworld_model_planner.txt | on level 2+, right before every continuation/reset/stuck: a 7-shot pgroup (“hard refactoring”) — 3 engine steps + 3 state_io steps + 1 planner step of compression/generalization orders |
on_new_level_v1.txt | the first iteration after the level index changes: orders a full-element inventory of the new level’s first frame (classified known / new / uncertain) |
death_prompt.txt | sent first on entering reset_protocol via GAME_OVER: “fold why you died into the model; inspecting the animation PNGs is mandatory” |
trouble1_prompt.txt / trouble1_prompt_level1.txt | when reset after burning 100+ steps on the same level since the last trouble1 |
trouble2_prompt.txt / trouble2_prompt_level1.txt | when reset after burning 200+ steps — a fresh Codex session is created and rebooted with main_prompt + trouble2 (“you may have tunnel vision”) |
stuck_reminder_prompt.txt | when a whole iteration passes with zero game steps: “if you don’t move, you lose.” If still no progress, stop_condition ends the run |
recovery_prompt.txt | on a --recovery-mode restart: orders a review of model, logs, and game state plus a situation report |
reset_prompt.txt, light_simplification.txt | the files exist but are absent from agent.py’s load_prompt call list — apparently unused on the current controller path (honesty notice) |
Stop conditions (stop_condition): current-level steps ≥ 1500,
no progress even after the stuck protocol, game cleared, or the current level cannot be found.
save_mismatch_region_png_v1/_raise_frame_mismatch are the
prototype of our engine’s mismatch-localization primitive.critique_prompt.md’s
adversarial generalization review (“are you hiding level-specific memorization?”) into the critic of our sleep phase.predict_next skills scored on the game’s own logged transitions
(Game.fork stays sealed), so no separate file layer is needed.