The behavior story
Astra finished all 15 September opening attempts without a recorded state-load request. DeepSeek V4.1 Flash finished eight of nine with a median of 41 loads. Both found ways through. What did they do when a move failed?
169 September and October opening attempts. The separate Astra marathon and its late reload appear in the marathon analysis.
A stalled position can prompt a better input, a measurement of the room, or another look at an old picture. These September examples connect the recorded action to what happened next.
HD recreation from retained run states. Textures by MasterKillua.
GPT-6 Astra · low effort · September 2026
A command can succeed without moving Mario
v3-codex-gpt6astra-low-tortle-001
HD recreation from retained run states. Textures by MasterKillua.
DeepSeek V4.1 Flash · low effort · September 2026
Reading the staircase out of the game
v3-opencode-ds41flash-low-mander-001
HD recreation from retained run states. Textures by MasterKillua.
GLM-5.3-Flash · low effort · September 2026
Reading the same two pictures
v3-opencode-glm53flash-low-tortle-001These scenes are recreated with HD textures from retained saved states and recorded inputs. The original captures are linked below each image. Actions and outcomes are checked against transcripts and scorer records.
20 traces, one route
Similar checkpoint scores can hide different routes and recoveries. Compare recorded movement inside the castle or switch to the full checkpoint clock.
October examples include the fastest GPT-6.1 Sol and GPT-6 Sol finishes and a GPT-6 Luna attempt that reached the castle but no later checkpoint. September and August examples also include both finishes and stalls. The human used a direct controller; these are individual runs, not model averages.
Only observed castle gameplay with control enabled is plotted.
Time covers retained segments, not total active play. See the filtering details below.
Both plots share a coordinate scale; height over time appears below. Lines stop at excluded samples and room changes.
| Segment | Run clock | x | y | z |
|---|---|---|---|---|
| 1 | 9:20 | 0 | 0 | 480 |
| 1 | 9:28 | 0 | 0 | 480 |
| 1 | 9:36 | -91.8 | 0 | 257.1 |
| 1 | 9:44 | -91.8 | 0 | 257.1 |
| 1 | 9:52 | -272.9 | 0 | 343.6 |
| 1 | 10:00 | -272.9 | 0 | 343.6 |
| 1 | 10:07 | -334.1 | 0 | 118.6 |
| 1 | 10:15 | -334.1 | 0 | 118.6 |
| 1 | 10:24 | -422.2 | 0 | -44 |
| 1 | 10:32 | -422.2 | 0 | -44 |
| 1 | 10:41 | -422.6 | 60 | -260.7 |
| 1 | 10:49 | -422.6 | 60 | -260.7 |
| 1 | 10:58 | -422.6 | 60 | -260.7 |
| 1 | 11:06 | -400.9 | 80 | -293.1 |
| 1 | 11:15 | -299.5 | 100 | -344 |
| 1 | 11:22 | -299.5 | 100 | -344 |
| 1 | 11:31 | -40.5 | 100 | -336.2 |
| 1 | 11:38 | -40.5 | 100 | -336.2 |
Run sampled about every 8.2s (median).
| Segment | Run clock | x | y | z |
|---|---|---|---|---|
| 1 | 10:27 | -71.4 | 0 | 440.8 |
| 1 | 10:35 | -71.4 | 0 | 440.8 |
| 1 | 10:43 | -71.4 | 0 | 440.8 |
| 1 | 10:50 | -200.9 | 0 | 398 |
| 1 | 10:58 | -265.7 | 0 | 376.6 |
| 1 | 11:06 | -468.5 | 0 | 225.5 |
| 1 | 11:13 | -468.5 | 0 | 225.5 |
| 1 | 11:21 | -476.2 | 0 | 20.9 |
| 1 | 11:28 | -487 | 10 | -60.3 |
| 1 | 11:36 | -400 | 50 | -135.5 |
| 1 | 11:44 | -382.8 | 50 | -170.2 |
| 1 | 11:51 | -330.2 | 100 | -329.4 |
| 1 | 11:59 | -297.3 | 100 | -396.7 |
| 1 | 12:07 | -236.5 | 100 | -402.9 |
| 1 | 12:15 | -236.5 | 100 | -402.9 |
| 1 | 12:23 | -236.5 | 100 | -402.9 |
| 1 | 12:31 | -83.4 | 100 | -387 |
| 1 | 12:38 | -83.4 | 100 | -387 |
| 1 | 12:46 | -83.4 | 100 | -387 |
| 1 | 12:55 | -83.4 | 111.1 | -387 |
| 1 | 13:04 | -83.4 | 100 | -387 |
| 1 | 13:12 | -83.4 | 100 | -387 |
| 1 | 13:20 | 27 | 100 | -380.1 |
| 1 | 13:28 | 27 | 100 | -380.1 |
Run sampled about every 8.2s (median).
Circle: first sample. Square: last sample. The paths show sampled coordinates, not room maps.
Samples require a normal context and enabled input in one of four castle gameplay rooms. Exclusions break the plotted lines. Time estimates sum intervals with two consecutive eligible endpoints in the same room; control can change between polls.
External scorer checkpoint times include the intro and cutscenes. Repeated-position intervals allow no sampling gap above 10 seconds. Height charts share a scale and use time since the first retained observation, including excluded gaps. Sparse samples cannot establish a continuous path or explain a delay.
Where available, original figures are unfiltered archival downloads and include cutscene positions:
Two August attempts show the difference between repeated travel and repeated inputs.
A Gemini 3.5 Flash attempt recorded 43 map transitions across nine distinct maps. About 81% revisited a map it had already seen. Its scored progress stopped at G3. The repeated travel is visible; the transition count alone does not explain each detour.
Gemini 3.5 profile
A Gemini 3.6 Flash attempt spent about 39 minutes at one horizontal position inside the entry hall, jumping through scripted A presses. It later spent about 30 minutes at another position. The script continued without reliably checking whether its presumed scene transition had happened.
Gemini 3.6 profileThe model profiles
Explore 181 August and 169 September and October attempts in separate profiles. All finish counts refer to entry into the opening Bowser battle. Reading the evidence explains the scope.
| Condition | Period | G4 / attempts | Finish time |
|---|---|---|---|
| Astra | September | 15 / 15 | 24:58 · median |
| Fable 5.1 | September | 12 / 15 | 35:58 · median |
| DeepSeek V4.1 Flash | September | 8 / 9 | 62:35 · median |
| Gemini 3.8 Flash | September | 6 / 9 | 77:14 · median |
| DeepSeek V4 Flash Vision | September | 0 / 9 | no finishes |
| GLM-5.3-Flash | September | 0 / 9 | no finishes |
| Opus 5 | September | 3 / 3 | 34:23 · median |
| Opus 4.8 | September | 2 / 3 | 76:27 · median |
| Opus 4.7 | September | 6 / 15 | 44:42 · median |
| Opus 4.6 | September | 1 / 12 | no median (n<2) · median |
| Opus 4.5 | September | 0 / 9 | no finishes |
| GPT-6.1 Sol | October | 15 / 15 | 26:25 · median |
| GPT-6 Sol | October | 15 / 15 | 34:32 · median |
| GPT-6 Luna | October | 0 / 15 | no finishes |
| Opus 5.5 | September | 16 / 16 | 28:02 · median |
| Opus 5 | August | 12 / 12 | 36:01 · median |
| GPT-5.6 Sol | August | 15 / 15 | 50:10 · median |
| Opus 4.8 | August | 10 / 12 | 53:34 · median |
| Fable 5 | August | 9 / 12 | 38:59 · median |
| GLM 5.2 | August | 3 / 6 | 67:04 · median |
| Kimi K3 · Droid | August | 4 / 9 | 49:15 · median |
| GPT-5.5 | August | 5 / 12 | 57:45 · median |
| GPT-5.6 Terra | August | 5 / 15 | 60:23 · median |
| Sonnet 5 | August | 4 / 12 | 76:52 · median |
| GPT-5.6 Luna | August | 4 / 15 | 78:00 · median |
| Grok 4.6 | August | 3 / 12 | 71:58 · median |
| Gemini 3.7 Flash | August | 2 / 9 | 70:19 · median |
| Gemini 3.1 Pro | August | 1 / 6 | 47:53 · single finish |
| Grok 4.5 | August | 1 / 9 | 58:52 · single finish |
| Gemini 3.6 Flash | August | 0 / 9 | never finished |
| Gemini 3.5 Flash | August | 0 / 9 | never finished |
| Kimi K2.7 Code | August | 0 / 3 | never finished |
| Kimi K3 · OpenCode | August | 0 / 3 | never finished |
| Kimi K3 · Cursor | August | 0 / 1 | never finished |
Fifteen finishes, with failures turned into small control experiments.
All fifteen attempts finished, with checkpoints saved but no recorded load requests. Successful runs used both memory-assisted navigation and a screenshot-guided route. The common useful move was checking whether an input changed the game before deciding what to try next.
Two upper-left holds left the position unchanged. Slightly reducing the diagonal stick input produced motion; the run went on to finish in 21m 11s.
low effortRun: v3-codex-gpt6astra-low-tortle-001After suspecting a guest obstruction, it measured full-diagonal input as zero and reduced input as nonzero, then reached the staircase and completed in 30m 21s.
max effortRun: v3-codex-gpt6astra-max-saur-001A short action runner, screenshots and detours around a guard and sofa were enough for a 22m 14s finish; no agent-authored RAM reader was found in its command stream.
low effortRun: v3-codex-gpt6astra-low-mander-002A dialogue batch ended unexpectedly. It inspected the paused emulator and resumed with smaller batches, finishing in 26m 36s.
xhigh effortRun: v3-codex-gpt6astra-xhigh-saur-001Every effort finished 3/3. Completion time did not rise or fall consistently with effort.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 0 of 15 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| lowcodex · 9b8e68a | 3/3 | 55 | 0 |
| mediumcodex · 9b8e68a | 3/3 | 50 | 0 |
| highcodex · 9b8e68a | 3/3 | 57 | 0 |
| xhighcodex · 9b8e68a | 3/3 | 57 | 0 |
| maxcodex · 9b8e68a | 3/3 | 70 | 0 |
Twelve finishes; the three misses ended while waiting for background play.
Twelve attempts finished; three native sessions ended after promising to resume from background notifications. Background play also succeeded when a monitor stayed active. Navigation detours and tool handoffs both affected these runs, so the misses cannot all be explained as difficulty with the castle.
It restored a checkpoint for two guard detours. The first remained in map 0; the second entered map 1. It finished in 21m 58s, the fastest Fable run.
medium effortRun: v3-claude-fable51-sub-medium-mander-002Repeated dialogue batches only made Mario jump in place. It stopped them, looked up the positional scene trigger, walked to it and finished in 40m 29s.
medium effortRun: v3-claude-fable51-sub-medium-tortle-001It tried a five-conversation strategy, reconsidered the route, restored a prior state and identified the hall with a memory scan. It eventually finished in 53m 24s.
low effortRun: v3-claude-fable51-api-low-mander-001After reaching the upper hall, it launched a walk-to-Peach and dialogue chain in the background, promised to await notification and exited at 38m 02s without the battle gate.
max effortRun: v3-claude-fable51-sub-max-tortle-001Two research agents supplied source facts while a monitored driver advanced play. It tightened noisy monitor filters, checked the scene and finished in 34m 19s.
max effortRun: v3-claude-fable51-sub-max-mander-002Low, medium and high finished 3/3; xhigh 2/3; max 1/3. One low run used the API route; the others used the subscription route. Service and host differences limit the comparison.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 6 of 15 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| lowclaude · a6deb5f | 1/1 | 9 | 4 |
| lowclaude · 04c3110 | 1/1 | 9 | 1 |
| lowclaude · fe982c4 | 1/1 | 9 | 3 |
| mediumclaude · 025487f | 1/1 | 7 | 0 |
| mediumclaude · 04c3110 | 1/1 | 6 | 0 |
| mediumclaude · b49e0fe | 1/1 | 13 | 4 |
| highclaude · 75c2219 | 1/1 | 12 | 26 |
| highclaude · 8e766f4 | 1/1 | 14 | 13 |
| highclaude · e313c87 | 1/1 | 8 | 0 |
| xhighclaude · bba8c57 | 1/2 | 4 | 0 |
| xhighclaude · d774cbd | 1/1 | 8 | 0 |
| maxclaude · d774cbd | 1/3 | 4 | 0 |
Seven of fifteen runs recorded direct native delegation, with nine requests in total. That count excludes delegation hidden inside scripts or workflows.
Eight finishes from nine attempts, with several breakthroughs coming when it replaced uncertain navigation with measured geometry.
Direction sweeps, checkpoints and geometry readers supplied different ways around obstacles. Eight attempts finished, including successes with 114 reload requests and with none. The lone miss still recovered from a staircase search, but too late to complete the remaining route.
After direction sweeps failed to find the climb, it decoded collision triangles from live memory. It identified the west staircase, moved onto it, and checked Mario rising through successive heights to the balcony. The run finished in 56.6 minutes.
low effortRun: v3-opencode-ds41flash-low-mander-001It decoded the corridor geometry, located the far eastern door, and fed the resulting route into its waypoint walker. The next commands crossed the corridor and opened the onward door. It finished in 51.0 minutes without a reload.
max effortRun: v3-opencode-ds41flash-max-tortle-001A broad staircase search timed out after 30 minutes. A focused east-side climb finally worked, but the run reached the next corridor only at 83.7 minutes, then aimed north instead of along the eastbound route. It ran out of time there.
max effortRun: v3-opencode-ds41flash-max-saur-003Low and high finished 3/3; max 2/3. The fastest finish was low effort at 44.6 minutes.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 8 of 9 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| lowopencode · 1418069 | 3/3 | 6 | 33 |
| highopencode · 1418069 | 2/2 | 17 | 86 |
| highopencode · a322fde | 1/1 | 9 | 42 |
| maxopencode · 1418069 | 2/3 | 8 | 16 |
Six finishes from nine counted attempts, with recoveries from awkward castle routes and two clean runs that reached the upper room too late.
Six attempts finished, including a recovery from a bedroom detour and a finish just before the cap. Two valid misses reached the upper room too late. A separate attempt issued a forbidden memory write and remains an invalid attempt in the denominator.
It combined a cutscene-advance helper with direct stair and door adjustments, including moving around the guard to reach the doorway. A final scene runner carried the attempt into Bowser at 36.4 minutes, the fastest Gemini finish in this sample.
high effortRun: v3-agy-gemini38flash-api-high-saur-002It spent part of the castle run opening and exploring Peach's bedroom, then returned to the hall. Further stair and door attempts finally put it on the upper route. It finished at 84.0 minutes, showing that a long wrong turn did not always end the run.
low effortRun: v3-agy-gemini38flash-api-low-saur-006After working through the castle doors, it launched a script to advance the final scene and kept checking while it ran. The external scorer detected Bowser at 89 minutes 47.2 seconds, only 12.8 seconds before the nominal 90-minute limit.
medium effortRun: v3-agy-gemini38flash-api-medium-tortle-002It reached the castle hall through controller play, struggled with the staircase, and then issued a game-memory write at 83.1 minutes. That action invalidated the attempt. Its later actions do not supply a recovery story.
medium effortRun: v3-agy-gemini38flash-api-medium-saur-001Low, medium and high each finished 2/3. The medium denominator includes the invalid attempt.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 4 of 9 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| lowagy · 4642768 | 0/1 | 5 | 1 |
| lowagy · 8e766f4 | 1/1 | 7 | 0 |
| lowagy · d4aa892 | 1/1 | 19 | 5 |
| mediumagy · 4642768 | 2/3 | 8 | 0 |
| highagy · 4642768 | 2/3 | 8 | 0 |
The denominator includes the RAM-write violation; only progress recorded before the write receives credit.
All nine attempts reached the castle, but none reached Bowser; the central difficulty was converting a visible staircase or door into a working route.
All nine attempts reached the castle and none finished. Several diagnosed a route as inaccessible after failed direction tests. One late run instead replaced a direct line with waypoints and reached the upper room. Reading geometry helped only when it produced a workable next move.
It tried full-angle sweeps, height and floor checks, jumping, alternate movement inputs and backtracking. It repeatedly concluded that the staircase could not be climbed with the available tools, then continued probing. The scorer recorded the castle hall, but no upper balcony.
high effortRun: v3-opencode-ds4flashvision-high-tortle-003A mistaken room sequence led it to the guard who opens Peach's bedroom. It pursued the conversation, entered the room and briefly called it a dead end, then returned to the idea that the story scene should happen there. It never reached the upper balcony gate.
low effortRun: v3-opencode-ds4flashvision-low-mander-003A direct line to the upper floor failed against the room's walls. Replacing it with west-side waypoints produced an actual climb and an onward door transition. It reached the upper room at 86.1 minutes, then prepared the final scene script without reaching Bowser before the cap.
max effortRun: v3-opencode-ds4flashvision-max-mander-002Low, high and max each finished 0/3. The furthest run was max effort, but the other two max attempts stopped earlier.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 9 of 9 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| lowopencode · 16ba6b6 | 0/3 | 9 | 85 |
| highopencode · 16ba6b6 | 0/2 | 25.5 | 83 |
| highopencode · 6ffc478 | 0/1 | 50 | 203 |
| maxopencode · 16ba6b6 | 0/3 | 7 | 17 |
All nine attempts reached the castle, but none finished. Some made late navigation breakthroughs; one spent most of its run rereading the same captured images.
The nine attempts lost time in different ways: misreading a playable scene, repairing position readers, searching for stairs, or rereading old captures. Some made late navigation breakthroughs. None finished, and the repeated-image case should not stand for every run.
Early mashing had already created the save and reached the party. After treating the scene as a stuck intro, it recognized that Mario was controllable and resumed navigation. It reached the upper room at 88.1 minutes, but not Bowser.
low effortRun: v3-opencode-glm53flash-low-mander-001Its position helper repeatedly changed as it investigated byte order and candidate addresses. When the authoritative address still appeared wrong, it checked the game version and discovered that it had added the position offset incorrectly. It eventually entered the castle hall at 83.4 minutes.
high effortRun: v3-opencode-glm53flash-high-tortle-002It first identified the castle using the wrong area names. After correcting that interpretation, it found the east staircase, reached the balcony and opened the onward door. The scorer recorded the corridor at 79.7 minutes; the run ended there.
max effortRun: v3-opencode-glm53flash-max-saur-001After repeated direction probes, it reloaded the hall entrance and alternated left and up inputs along the wall. Position bytes showed the first gain in height. A revised diagonal input then reached the balcony, and the scorer recorded the next corridor at 67.0 minutes. Later backtracking and door attempts did not produce a finish.
low effortRun: v3-opencode-glm53flash-low-saur-002After launching a party-scene script, it kept reading previously captured images. Two files were opened 384 times each across roughly 73 minutes. The remaining native tool stream contains image reads rather than further navigation, and the scorer never recorded the castle hall.
low effortRun: v3-opencode-glm53flash-low-tortle-001Low, high and max each finished 0/3. The furthest low run reached the upper room; high reached the balcony and max the corridor.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 4 of 9 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| lowopencode · 13cdf30 | 0/2 | 9.5 | 31.5 |
| lowopencode · f5f1a09 | 0/1 | 3 | 0 |
| highopencode · 13cdf30 | 0/2 | 1.5 | 1 |
| highopencode · 91054e5 | 0/1 | 5 | 5 |
| maxopencode · 13cdf30 | 0/2 | 6 | 1 |
| maxopencode · f5f1a09 | 0/1 | 2 | 0 |
Three finishes, with short direction tests and checkpoint restores turning stalled moves into workable routes.
All three September max-effort attempts finished between 32m 45s and 36m 06s. Each built small input and position helpers, checked movement against screenshots, and restored checkpoints when exploring alternatives. The shared pattern was local repair: find the stair entrance, get around an obstacle, or return from a wrong door, then continue.
Eight-direction probes and a second set of longer holds all ended at ground level. From the saved position, a 100-frame northward hold finally registered a climb. It followed the stairs, worked around the balcony obstacles and finished in 34m 23s.
max effortRun: v3-claude-opus5-sub-max-mander-001After reaching the balcony, it reloaded the hall entrance and tried the lower blue door. That detour did not advance the route. Restoring the balcony checkpoint and passing the guard from the south reached the onward door; the run finished in 36m 06s.
max effortRun: v3-claude-opus5-sub-max-saur-002Repeated direction tests left Mario at ground level. It backed south, moved farther west and then walked north while checking height. The next samples climbed from 0 to 50, then 100 after turning at the landing. It finished in 32m 45s.
max effortRun: v3-claude-opus5-sub-max-tortle-003Max finished 3/3. These September runs add a new effort setting; the August low-through-xhigh results remain a separate comparison.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 3 of 3 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| maxclaude · 6d3281d | 3/3 | 13 | 17 |
Knowing where the door was did not make the staircase easy: two late finishes and one hall timeout.
All three September max-effort attempts used research agents to look up game state or the castle route. They identified the onward door but repeatedly approached the staircase from places that kept Mario on the floor. Two eventually found the far-west approach and finished; the third kept searching until the time cap.
After entering the hall at 36m 37s, it tried direction sweeps, checkpoint restores and floor-collider readings. It corrected an offset error in that reader, but never recorded the upper balcony. The run ended at the cap with 53 reload requests.
max effortRun: v3-claude-opus48-sub-max-mander-001It treated the entrance landing door as unresponsive and returned to the ground floor. A source lookup supplied its position and press-A trigger, and a closer approach opened it. A further staircase search finally reached the balcony at 46m 19s; Bowser followed at 63m 51s.
max effortRun: v3-claude-opus48-sub-max-mander-002After repeated searches, it added floor-collider readings and swept north along the far-west edge. Height finally rose, and it followed the turning staircase onto the balcony at 67m 45s. It then navigated the remaining doors and reached Bowser at 89m 03s.
max effortRun: v3-claude-opus48-sub-max-mander-003Max finished 2/3, at 63m 51s and 89m 03s. These September attempts are separate from the August low-through-xhigh results.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 3 of 3 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| maxclaude · 76c2213 | 0/1 | 9 | 53 |
| maxclaude · 6d3281d | 2/2 | 11.5 | 24 |
High finished 3/3; lower efforts stopped early, while max included a timeout and a last-minute finish.
Successful runs corrected door approaches and sometimes restored checkpoints. The larger failure was ending the session before reaching Bowser: all six low and medium attempts returned before the 90-minute cap. One low-effort run mistook the opening objective for a full playthrough.
After reaching the balcony, it restored the hall checkpoint and searched the right side. A second restore returned it to the left stairs. It moved around a pillar, adjusted its approach to the guarded door and pressed A to reach the next room at 22m 24s. Bowser followed at 30m 39s.
high effortRun: v3-claude-opus47-sub-high-mander-001It entered the castle hall, then described the task as a full playthrough and said the final Bowser battle was hours away. It returned a progress summary at 38m 58s without reaching the opening Bowser battle; the 90-minute allowance had not expired.
low effortRun: v3-claude-opus47-sub-low-tortle-003Repeated approaches and checkpoint restores kept returning it to familiar castle rooms. Near the end, an A press by the guard still left it in the stairs room. A further rightward adjustment and another A press opened the next door. It reached the upper room at 83m 41s and Bowser at 89m 34s, 26 seconds before the nominal cap.
max effortRun: v3-claude-opus47-sub-max-tortle-003Low 0/3, medium 0/3, high 3/3, xhigh 1/3, max 2/3. High finished in 30m 39s–43m 48s; max finished at 45m 36s and 89m 34s. These small samples do not show a monotonic benefit from increasing effort.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 10 of 15 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| lowclaude · 6d3281d | 0/1 | 1 | 1 |
| lowclaude · 63835a4 | 0/2 | 0 | 0 |
| mediumclaude · 63835a4 | 0/3 | 3 | 0 |
| highclaude · 63835a4 | 3/3 | 5 | 5 |
| xhighclaude · 63835a4 | 1/3 | 5 | 4 |
| maxclaude · 63835a4 | 2/3 | 6 | 10 |
One of twelve runs reached the scored battle: a max-effort run finished in 75m 14s.
Door and stair searches repeatedly returned to familiar castle rooms. It tried different movement angles, checked positions and restored checkpoints, but often revisited routes without finding the next exit. Its one completion spent nearly 48 minutes in the entrance room before opening the hall door, then reached the battle about 17 minutes later.
After repeated stair searches and checkpoint restores, it pressed A while moving toward the entrance room’s upper door. The hall checkpoint followed at 57m 49s. It crossed the remaining castle rooms and reached the scored battle at 75m 14s.
max effortRun: v3-claude-opus46-sub-max-tortle-003It reached the hall’s upper balcony at 44m 34s, then continued trying door and stair variations across earlier rooms. The command log records 64 checkpoint-load requests. It finished back in the entrance room without reaching the next stairs checkpoint.
high effortRun: v3-claude-opus46-sub-high-herb-002Unable to leave the castle entrance room, it suspected its savestates and stopped the emulator at 86m 15s, then started it again. The remaining time replayed the opening sequence; no later castle checkpoint was recorded.
low effortRun: v3-claude-opus46-sub-low-mander-001Low 0/3, medium 0/3, high 0/3, max 1/3. All twelve reached the castle; eleven ran to the time cap. Max produced the only completion, but three runs per setting do not establish an effort effect.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 12 of 12 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| lowclaude · 63835a4 | 0/3 | 5 | 14 |
| mediumclaude · 63835a4 | 0/3 | 5 | 15 |
| highclaude · 63835a4 | 0/3 | 11 | 38 |
| maxclaude · 6e8d4ea | 1/3 | 7 | 26 |
All nine reached the castle; none entered the scored battle. Six returned unfinished progress reports before the time limit.
It repeatedly changed movement angles, revisited doors and tried more guard dialogue. Some corrections produced real progress: two runs reached the next stair room. Both then searched familiar rooms without finding the onward route.
After suggesting the hall stairs might be decorative, it approached from the far left and changed direction while climbing. The upper balcony checkpoint followed at 44m 31s and the next stair room at 49m 29s. It kept searching around a star door, then returned a progress report asking whether to continue. The run stopped at 65m 23s without the battle.
high effortRun: v3-claude-opus45-sub-high-saur-002It reached the stairs checkpoint at 36m 8s, then returned to the hall and tried talking to guests and the guard. One later command sent 150 A presses over about five minutes. It then reported that the guard was still blocking and started another dialogue loop. The scorer reached the time cap with no further checkpoint.
low effortRun: v3-claude-opus45-sub-low-mander-001Low 0/3, medium 0/3, high 0/3. One low, all three medium and two high runs stopped early; the other three hit the time cap. Low and high each reached the stairs checkpoint once. Max is not included.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 9 of 9 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| lowclaude · 63835a4 | 0/3 | 5 | 3 |
| mediumclaude · 75205ed | 0/3 | 5 | 7 |
| highclaude · 75205ed | 0/3 | 7 | 3 |
Finished all 15 attempts; only one requested a state reload.
GPT-6.1 Sol reached the opening Bowser battle in all 15 attempts. The median completed attempt took 26m 25s, and finishes ranged from 19m 32s to 37m 01s. The median attempt recorded 68 screenshot requests, 48 save requests and no load requests; one attempt recorded a single load request. Checkpoints were frequent, but returning to one was rare.
In the high-effort example, Sol added memory reads for the current room and Mario’s position to its input script, then used height readings to confirm a staircase climb in the castle hall. A script checked the game mode after each input segment and stopped at the battle. The scorer recorded the stairs gate at 13m 29s and the opening battle at 19m 32s. This was its fastest finish, not a typical time.
The tool stream records memory reads for the current room and Mario’s coordinates, then raw stick inputs followed by height checks on the castle-hall staircase. A script checked the game mode after each input segment. The scorer recorded the stairs gate at 809.4 seconds and G4 at 1171.6 seconds. The command ledger recorded 61 screenshot requests and no state-load requests.
high effortRun: v3-codex-gpt61sol-high-mander-003The approach first left Mario under a side staircase at floor level. The native command stream then records collision-data lookups and coordinate-targeted movement before the climb. The scorer recorded the castle hall at 883.5 seconds and the upper hall at 1501.5 seconds, with sampled telemetry in the upper hall throughout; G4 followed at 2220.6 seconds, the slowest of 15 finishes. No state-load request appears in its command ledger.
max effortRun: v3-codex-gpt61sol-max-mander-002All five effort settings produced three completions each. This profile describes the same 15 attempts the leaderboard uses. Timing differences between settings are descriptive; three attempts per setting do not isolate effort from run variation.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 1 of 15 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| lowcodex · fe60b0d | 3/3 | 46 | 0 |
| mediumcodex · fe60b0d | 3/3 | 58 | 0 |
| highcodex · fe60b0d | 3/3 | 29 | 0 |
| xhighcodex · fe60b0d | 3/3 | 54 | 0 |
| maxcodex · fe60b0d | 3/3 | 51 | 0 |
Finished all 15 October reruns, usually without loading a saved state.
GPT-6 Sol reached the opening Bowser battle in all 15 October reruns. The median completed attempt took 34m 32s, and finishes ranged from 22m 10s to 71m 36s. The median attempt recorded 110 screenshot requests, 27 save requests and no load requests; two command logs recorded load requests.
In the low-effort example, Sol alternated short analog inputs with screenshots and used a script to advance the opening dialogue while saving checkpoints. When its first approach in the upper hall slipped under the next staircase, it repositioned and climbed without loading a saved state. The scorer recorded the stairs gate at 13m 22s and the opening battle at 22m 10s. This was the fastest of the 15 finishes, not a typical time.
After the first approach in the upper hall slipped under the next staircase, the transcript records a repositioning at its lower end followed by directional inputs and fresh screenshots. No state-load request appears in the command ledger. The scorer recorded the upper hall at 708.6 seconds, the stairs gate at 801.6 seconds and G4 at 1329.7 seconds.
low effortRun: v3-codex-gpt6sol-postvision-low-mander-003The scorer recorded the stairs gate at 1675.4 seconds and the upper castle at 3047.6 seconds; sampled telemetry places nearly all of that interval in the castle stairs room. The transcript records a direction sweep and thirteen successive staircase-approach scripts before a height reading confirmed the climb, then balcony and door-alignment scripts. Five state-load requests came later, within 124 frames, while telemetry showed the Bowser approach room. G4 followed at 4296.1 seconds, the slowest of 15 finishes.
max effortRun: v3-codex-gpt6sol-postvision-max-mander-005All five effort settings produced three completions each. The slowest finish came at max effort and the fastest at low effort, but with three attempts per setting these are individual runs, not an effort effect. This profile describes the same 15 attempts the leaderboard uses.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 2 of 15 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| lowcodex · fe60b0d | 3/3 | 28 | 0 |
| mediumcodex · fe60b0d | 3/3 | 31 | 0 |
| highcodex · fe60b0d | 3/3 | 27 | 0 |
| xhighcodex · fe60b0d | 3/3 | 8 | 0 |
| maxcodex · fe60b0d | 3/3 | 27 | 0 |
Usually reached the castle, but none of 15 October reruns reached Bowser.
GPT-6 Luna did not reach the opening Bowser battle in any of its 15 October reruns. Eleven ended at the 90-minute limit and four ended when the agent stopped. Twelve reached at least the castle door and seven reached the castle hall or beyond; the deepest reached the Bowser approach. Ten command logs recorded state-load requests, with a median of two per attempt.
The default trace follows the median outcome. In the max-effort example, Luna reached the castle door at 24m 41s, and every later telemetry sample stayed in the castle entry hall for the remaining 65 minutes. Its scripts read Mario’s position and an input-disabled flag, found that unchanged coordinates came from an open dialogue, and used height readings while climbing the hall’s staircase. The scorer recorded no castle-hall gate before the time limit.
Final reports did not reliably describe progress. One high-effort attempt reported all four goals complete about 19 minutes in, when the scorer had recorded only file creation.
The scorer recorded the castle door at 1481.4 seconds and no later gate. All 512 later telemetry samples are in the castle entry hall. The transcript records position and input-flag reads, an open dialogue identified as the reason coordinates were unchanged, and height readings during a staircase climb, followed by repeated attempts at a door the agent described as guarded. The command ledger contains 177 screenshot requests and five state-load requests.
max effortRun: v3-codex-gpt6luna-postvision-max-mander-003After creating a file and reading the walkthrough, the agent reported that all four goals were complete. The scorer had recorded only file creation, at 869.8 seconds. Every later telemetry sample remained in Mario’s house with player input disabled until the attempt stopped at 1163.1 seconds.
high effortRun: v3-codex-gpt6luna-postvision-high-mander-003The scorer recorded the castle hall at 2425.7 seconds and the Bowser approach at 5037.2 seconds. The transcript records the agent still advancing pre-battle dialogue when the 90-minute limit ended the run; no G4 was recorded. The command ledger contains 257 screenshot requests and 25 state-load requests.
max effortRun: v3-codex-gpt6luna-postvision-max-mander-002No effort setting produced a completion. All three low-effort attempts ended when the agent stopped, two at the castle door and one before any scored gate. Medium, high and extra-high effort each had two attempts reach the castle hall or beyond; at max effort one did, and it reached the Bowser approach. With three attempts per setting these are descriptive observations, not an effort effect.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 10 of 15 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| lowcodex · fe60b0d | 0/3 | 1 | 0 |
| mediumcodex · fe60b0d | 0/3 | 52 | 2 |
| highcodex · fe60b0d | 0/3 | 47 | 0 |
| xhighcodex · fe60b0d | 0/3 | 53 | 5 |
| maxcodex · fe60b0d | 0/3 | 51 | 5 |
Finished all retained attempts; most did not request a state reload.
Opus 5.5 reached the opening Bowser battle in all 16 retained attempts. Four command logs recorded state-load requests. The median attempt recorded 74 screenshot requests, 8.5 save requests and no load requests; the median completed attempt took 28m 02s.
In the low-effort example, repeated inputs near an upper-hall guard did not immediately open the door. Opus read Mario’s live coordinates, changed its approach and crossed into the next room. The scorer recorded the stairs gate at 10m 38s and the opening battle at 17m 51s. This was its fastest retained finish, not a typical-run estimate.
The native Claude ledger recorded three direct delegation requests across three attempts. Most retained attempts had no such request; this inventory does not count work delegated inside shell scripts or infer what each delegate accomplished.
The tool stream records live coordinate reads followed by directional inputs around the upper-hall guard. The scorer recorded the stairs gate at 638.3 seconds and G4 at 1071.4 seconds. No state-load request appears in this attempt’s command ledger.
low effortRun: v3-claude-opus55-sub-low-herb-003All five effort settings produced completions. This profile describes 16 retained attempts; the leaderboard uses 15, three at each effort. Observed timing differences are descriptive and do not isolate effort from run variation or harness conditions.
Received controller commands, including harness traffic; these are requests rather than confirmed actions or individual model tool calls. Successful runs end sooner.
Medians across this model's runs. 4 of 16 runs recorded a load request.
| Effort / CLI | Finished | Save median | Load median |
|---|---|---|---|
| lowclaude · 0595b2b | 1/1 | 15 | 1 |
| lowclaude · 548c827 | 2/2 | 6.5 | 0.5 |
| mediumclaude · 0595b2b | 1/1 | 6 | 0 |
| mediumclaude · 548c827 | 2/2 | 5.5 | 0 |
| highclaude · 0595b2b | 4/4 | 8.5 | 0 |
| xhighclaude · 0595b2b | 1/1 | 16 | 1 |
| xhighclaude · 548c827 | 2/2 | 12 | 0 |
| maxclaude · 0595b2b | 1/1 | 13 | 0 |
| maxclaude · 548c827 | 2/2 | 7 | 0 |
Small scripts and repeatable probes, with little visible commentary.
How it played
All twelve attempts finished. The visible record often carries the plan in helper scripts: camera-aware walkers, wall sweeps and checks for battle entry. Eight attempts recorded no load request; four reused a saved position to compare candidate headings. That made a failed move testable: restore the position, change the heading, check whether Mario gained height. Some thinking was redacted, so sparse commentary does not measure how much reasoning occurred.
“Closed-loop walker: compensates for camera-relative stick mapping.”
from the session log of v3-opus5-high-mander-001
Finished by effort: low 3/3; medium 3/3; high 3/3; xhigh 3/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Median finish by effort: high 54:53 · low 26:52 · medium 28:43 · xhigh 43:13 (cells with fewer than two completions get no median)
Related profile:
Fifteen finishes, with checkpoints used mostly as insurance.
How it played
Sol often wrote a short script, ran it, checked the screenshot or telemetry, and chose the next input from the result. One run restored a shared state before four direction probes; another checked the frame counter when a long batch appeared hung. Eleven of fifteen attempts recorded no load request, while four saved more than seventy times. This was not a universal memory-led recipe: several successful runs relied on screenshots.
“I’m branching four cardinal 30-frame probes from one savestate so the working departure direction is established mechanically.”
from the session log of v3-codex-gpt56sol-max-mander-001
Finished by effort: low 3/3; medium 3/3; high 3/3; xhigh 3/3; max 3/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Median finish by effort: high 50:10 · low 37:13 · max 52:07 · medium 48:46 · xhigh 42:09 (cells with fewer than two completions get no median)
Related profile:
A hypothesis, a probe, then a revised explanation.
How it played
The ten completed runs repeatedly describe why a move failed before trying another: a wall, a landing below the stairs, or a guard intercepting a door press. One escalated from direction guesses to reading the floor collider. Nine of twelve attempts recorded loads, including checkpoint-based height probes. Opus 4.8 requested research agents in eleven attempts; Opus 5 requested none in twelve with the same native tool inventory. That is a contrast in this sample, not a measured benefit of delegation.
“I've blown this room's budget. Let me escalate to a decisive instrument: read which floor collider Mario is standing on, so I can tell stairs from floor mechanically.”
from the session log of v3-opus48-low-tortle-001
Finished by effort: low 3/3; medium 2/3; high 3/3; xhigh 2/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Median finish by effort: high 41:18 · low 82:59 · medium 45:57 · xhigh 52:45 (cells with fewer than two completions get no median)
Related profile:
Nine finishes; all three misses ended after promising to await background events.
How it played
Successful runs turn obstacles into small experiments: move around the guard, compensate for a rotated camera, or restore one position before probing six directions. Six of the nine finishers recorded no loads; the other three used one, one and six. The three failed native sessions ended after saying they would resume from background monitor events. Those recorded handoffs belong in the explanation of the misses; their exact scheduler cause has not been established.
“Let me stop guessing at the camera and probe systematically”
from the session log of v3-fable5-high-mander-003
Finished by effort: low 3/3; medium 3/3; high 2/3; xhigh 1/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Median finish by effort: high 47:49 · low 38:59 · medium 43:43 · xhigh n=1 (cells with fewer than two completions get no median)
Related profile:
Described screenshots and measured positions, with frequent retries.
Condition note: This endpoint takes no images, so sight came from two tools we built for it: a describe call backed by qwen3.6-35b and a LocateAnything-3B locator, served from a dedicated RTX 5090 vision host that never ran evals.
How it played
The model endpoint could not take images. In the three completed runs, separate vision tools described screenshots while the agent checked positions and tried another input. Checkpoints supported direction experiments, with seven to twenty-five load requests across the six attempts. A failed staircase run also repeatedly restored its position and changed direction, so using that method did not guarantee a climb. The visual sidecar is part of this condition, not evidence of native image understanding.
“So the whole run is really just the OPENING.”
from the session log of v3-glm52-high-metapod-004
Finished by effort: high 2/3; max 1/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Median finish by effort: high 67:14 · max n=1 (cells with fewer than two completions get no median)
Related profile:
Four finishes, usually correcting mistakes without reloading.
How it played
The four completed runs built position readers and, in one case, a walker that corrected movement from measured coordinate changes. Eight of nine attempts recorded no load request; the ninth recorded one. Kimi K3 finished four of nine on Droid, none of three on OpenCode and none of one on Cursor. The selected Cursor run reached the castle sooner than the selected Droid finisher, then stopped earlier. Effort, service and CLI conditions differ, so this is not a controlled harness comparison.
“This manual navigation is burning lots of cycles.”
from the session log of v3-kimik3-high-tortle-011
Finished by effort: low 2/3; high 1/3; max 1/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Median finish by effort: high n=1 · low 45:55 · max n=1 (cells with fewer than two completions get no median)
Related profile:
Successful runs switched from visual guesses to measured navigation.
How it played
The five completed runs begin with documentation and source reading, then encounter castle doors, stairs and guards that resist visual steering. One reloaded a clean lobby checkpoint after deciding its current position was unhelpful. Another noticed that starting a separate adapter for every tiny input was expensive and changed its batching. These are examples from finishers; they do not describe all seven incomplete attempts.
“This has become a bad local position. I’m reloading the clean lobby checkpoint at slot 6”
from the session log of v3-gpt55-high-mander-001
Finished by effort: low 0/3; medium 1/3; high 3/3; xhigh 1/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Median finish by effort: high 65:50 · medium n=1 · xhigh n=1 (cells with fewer than two completions get no median)
Related profile:
Inventive recoveries in successful runs, alongside several early endings.
How it played
The five completed runs include collision-mesh navigation, checkpoint-based direction sweeps and a search for a door plane. One used rising player height to test whether a staircase approach worked. In a retained narration, another diagnosed a stuck castle scene and reloaded a pre-Peach checkpoint; that records the agent’s diagnosis, not independent proof of a game fault. Seven of ten misses ended before the time ceiling, so long navigation struggles do not explain every failure.
Finished by effort: low 0/3; medium 0/3; high 1/3; xhigh 2/3; max 2/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Median finish by effort: high n=1 · max 64:12 · xhigh 52:25 (cells with fewer than two completions get no median)
Related profile:
Visual trial and error, plus one directly recorded background-task timeout.
How it played
The four completed runs use screenshots to interpret doors, guards and camera changes, sometimes switching to position reads when visual probing stalls. Nine of twelve attempts requested research agents, with twenty-eight requests in total. One low-effort attempt handed play to a background agent in its first minute; the harness later logged its 600-second background-wait ceiling and terminated the task. That ending is more specific than a model simply deciding to stop.
“I've made confirmed progress and hit a genuine navigation wall.”
from the session log of v3-sonnet5-low-saur-002
Finished by effort: low 1/3; medium 0/3; high 2/3; xhigh 1/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Median finish by effort: high 76:52 · low n=1 · xhigh n=1 (cells with fewer than two completions get no median)
Related profile:
Repeated visual probing; six misses shared the S1u checkpoint endpoint.
How it played
The four completed runs describe a guard intercepting door interactions and a landing that trapped Mario against a wall. Across all fifteen attempts, five recorded no save request and one incomplete run took 269 screenshots. Six of eleven misses ended at S1u; connecting those endpoints to the guard seen in successful runs remains an inference. Eight misses reached the time cap and three ended as model stops.
“I’ve confirmed the castle’s main stairs are navigable, but the upper landing is a wraparound corridor and the first long push pinned against its wall.”
from the session log of v3-gpt56luna-high-saur-001
Finished by effort: low 0/3; medium 1/3; high 2/3; xhigh 1/3; max 0/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Median finish by effort: high 60:42 · medium n=1 · xhigh n=1 (cells with fewer than two completions get no median)
Related profile:
Frequent screenshots and checkpoint retries, including a damaged checkpoint.
How it played
The three completed runs combine research, screenshots and memory checks. One discovered that a door needed Mario to press into the wall before interacting; another used collision reads to locate stairs. At least one unstick loop overwrote a useful saved position. Across twelve attempts, a single xhigh run recorded 396 loads. Grok 4.6 finished three of twelve compared with Grok 4.5’s one of nine; the extra checking did not make every attempt reliable.
“The interact prompt appeared, but A was pressed before Mario was pushing into the wall.”
from the session log of v3-cursor-grok46-high-mander-002
Finished by effort: low 0/3; medium 1/3; high 2/3; xhigh 0/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Median finish by effort: high 70:45 · medium n=1 (cells with fewer than two completions get no median)
Related profile:
Two finishes through background navigation and repeated route adjustments.
How it played
Both completed runs launched navigation or dialogue tasks and reviewed the results. One repeated short staircase status messages; those messages alone do not show what caused the climb to work. The other used timed guard conversations and searched memory for the room identifier when a door would not open. Seven other attempts stopped before battle entry. Their checkpoint records show varied depths, not one shared diagnosis.
“Climbing the ballroom stairs up to the balcony double doors.”
from the session log of v4-agy-gemini37flash-medium-mander-001
Finished by effort: low 0/3; medium 1/3; high 1/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Related profile:
One visual completion, with five misses at different checkpoints.
How it played
The sole finisher used screenshots, short direction probes and longer cutscene scripts. It recognized that Mario had fallen off a raised red carpet in the final hallway. Its corrected command record contains no agent-issued RAM reads. The remaining five attempts stopped at different checkpoints; one logged seventy-four loads and another a long unchanged-position interval. The successful visual interpretation does not explain those separate misses.
“The diagnostic showed that Mario was stuck against the wall under the stairs. Pushing down (towards the camera) successfully freed him onto the red carpet.”
from the session log of v3-gemini31pro-high-tortle-002
Finished by effort: low 0/3; high 1/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Related profile:
One finish amid heavy checkpoint traffic.
How it played
One low-effort attempt recorded 954 loads against 139 saves. The sole completed run used saved positions to retry stairs and doors, checked memory around transitions, and corrected an assumption that Mario was controllable while Luigi was still talking. It also changed held A inputs to separate press-and-release pulses. That successful run shows a purpose for some checkpoint traffic; it cannot explain every retry in the eight failures.
“Handed off mid-intro (Luigi still talking; stick cal invalid). Reloading and clearing dialogue before navigating.”
from the session log of v3-grok45-medium-tortle-001
Finished by effort: low 0/3; medium 1/3; high 0/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Related profile:
Repeated scripted inputs could consume most of a run without advancing the route.
How it played
One medium attempt remained at the same horizontal position inside the castle entry hall for about thirty-nine minutes while its scripted A presses made Mario jump. It later spent about thirty minutes at another position. The recorded plan treated the scene as something to advance through; the inputs did not reliably check that assumption. All nine attempts missed battle entry, but three progressed beyond G3. Time spent in the entry hall is not, by itself, time motionless.
Finished by effort: low 0/3; medium 0/3; high 0/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Backtracking and repeated visits, including a 43-transition attempt.
How it played
One low-effort run recorded forty-three map transitions across nine distinct maps; roughly eighty-one percent revisited a previously seen map. That is a specific example of repeated travel, not a minimum-route comparison. Other runs also spent heavily on saves and loads. The August audit records a forbidden memory write in this condition after its last credited gate; later activity is not evidence of legitimate progress. None of the nine attempts finished.
Finished by effort: low 0/3; medium 0/3; high 0/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Three attempts stopped at the same scored checkpoint.
How it played
All three high-effort attempts stopped at S1 and took 191–292 screenshots. Two recorded fifty-four and forty-one loads, respectively. Retained logs also show walkthrough-video research in one run. These observations establish active looking and retrying, but do not isolate a common cause for the three misses.
Finished by effort: high 0/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
Instrument debugging continued even when position traces showed little movement.
How it played
The three max-effort attempts missed battle entry: two stopped at G3 and one reached S1. A retained transcript shows collision-mesh inspection, a mistaken vertex layout and a revised decoder. Long unchanged-position intervals therefore do not establish that the agent was doing nothing. A synchronous research task also returned source facts that entered its telemetry loop. These conditions remain separate from the nine Droid and single Cursor attempts.
Finished by effort: max 0/3.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
One attempt, with button-registration and image-preview problems in its recorded diagnosis.
How it played
The retained transcript reports a corrected button mask, unreliable image previews and a collision dump to find the next door. It recorded three saves and no loads, then stopped at S1 without finishing. Its faster castle arrival than the selected Droid finisher did not translate into a deeper endpoint. One high-effort Cursor attempt cannot establish a general harness effect.
Finished by effort: high 0/1.
Archived all-attempt medians; the smaller figures refer to the 78 August finishers. Channel RAM counts include scorer polling. Unchanged-position intervals do not establish why Mario stayed still.
August profiles retain their original statistics and 78-finisher reference medians. September and October profiles use their own run counts and command summaries. Gate timing comes from the external scorer; collection periods are not pooled into a single behavioral baseline.
Among August finishers, Anthropic runs generally checked memory more often relative to screenshots than OpenAI runs. Sonnet was a counterexample: its four finishers used 0.2–0.9 agent RAM reads per screenshot.
35 Anthropic finishers, after subtracting scorer polling.
29 OpenAI finishers, after subtracting scorer polling.
Agent RAM reads per screenshot ordered 899 of the 1,015 cross-family finisher pairs in the same direction. This describes the sample; it does not establish which method caused a finish.
Among the 181 comparable August attempts, 27 requested native sub-agents, with 61 requests; 13 of those attempts finished. Opus 4.8 requested them in 11 of 12 attempts and Opus 5 in none of 12 under the same native inventory. The tested Codex inventory lacked native delegation, while Antigravity delegation could not be verified.
August’s retained narrative summaries cover its 78 finishers. Failed-run examples use checkpoint records and, where identified, retained native transcripts. A successful strategy does not describe every attempt by that model. Muse Spark is excluded from comparisons because of its image cap.
Most effort groups contain three attempts. Finish times exclude misses and do not isolate an effort effect. Single finishes are labeled individually; effort cells with fewer than two finishes have no median. August’s archived profile metrics include controller-channel traffic, including scorer RAM reads. The command comparison above subtracts that polling.
