Giving agents every chance to play
I was working on parallel-ish, a fork of the ParaLLEl N64 emulator that adds hi-res textures and improves scaling. I wanted agents to reach a room and check the rendering without turning my own playthrough into a debugging session. When I asked one to verify a change by playing Paper Mario, I expected checking the change to be the difficult part. Instead, I found myself coaching it through file select. Later, it could read a walkthrough telling it to find the door with a star, but struggled to reach the giant star-marked door right there on screen. Getting it to play became a project of its own.
Paper Mario Bench grew out of that surprise. It gives coding agents 90 minutes to play the opening, from the title screen to the start of the first Bowser battle. Along the way, they can inspect screenshots, read game memory, consult the decompiled source, and control the game with frame-counted inputs. The emulator pauses while they think, and a separate scorer measures their progress.
Pokémon and Slay the Spire have become examples of what agents can do with games. Anthropic's Fable 5 announcement featured both, demonstrating progress in visual understanding and memory. Paper Mario interests me because it combines those RPG decisions with movement and timing.
Its battles are turn-based, but they include timed action commands: little interactive challenges where choosing the right move is only part of succeeding. Outside combat, the agent has to navigate a 3D world with an analog stick, judge where Mario is, and coordinate movement with jumps, even just to get up the stairs. These are additional demands beyond selecting cards or navigating Pokémon's tile-based world. The opening benchmark exposes the navigation problem, while the marathons bring more of the combat and timing challenges into play.
We give agents tools to approach those challenges at their own pace, including savestates to retry a section. Can they understand what needs to happen, execute it, and recognize whether it worked?
Those controls grew out of watching agents press buttons without checking what happened, lose track of saves, and repeat actions that weren't helping. The harness page explains how the tools work and what agents are allowed to access.
Even with that help, the range of outcomes is striking. The comparison covers twenty-five models and 277 counted attempts, with 130 finishes. In September, Astra finished all fifteen opening runs and Fable 5.1 finished twelve of fifteen. DeepSeek V4 Flash Vision and GLM-5.3-Flash each had nine runs and no completions.
In August, Opus 5 finished all twelve selected runs and GPT-5.6 Sol finished all fifteen. September max-effort runs added three finishes out of three for Opus 5 and two out of three for Opus 4.8. Gemini 3.5 Flash, Gemini 3.6 Flash, and Kimi K2.7 Code had no completions. Some agents reached the castle quickly but couldn't make it up the stairs before the 90-minute cap. Astra now holds the fastest observed finish at 20 minutes and 5 seconds. The site labels collection periods and harness conditions alongside the results.
Anything below 100% on an individual run means the agent didn't complete the task. We give credit for partial progress because even getting partway through can be difficult. The standings show average and best route progress alongside completion counts, time, and estimated cost, so an average progress score should be read separately from the completion rate.
Watching how they get there is part of the appeal. Astra finished all fifteen September opening runs without a recorded state-load request. DeepSeek V4.1 Flash finished eight of nine with a median of 41 load requests. Frequent recovery can be useful, but it is not the only way through. Does the agent recognize that it is stuck? Does it try something useful, or keep repeating itself? The behavior page follows those differences, and the deep dive breaks down individual attempts, progress, and effort settings.
For the strongest models, the marathons are becoming the more interesting test. Once a model can finish the opening consistently, I want to see it continue into combat, navigation, and puzzles. Can it preserve what it has accomplished, recover from mistakes, and keep making progress over hours?
GPT-5.5's early Chapter 1 run needed coaching and disclosed assists while we developed the tools. Later evals and marathons had those tools from the start. The two September marathons reached the opening endpoint less than thirty seconds apart, then diverged over the remaining hours. Both used medium effort. Fable 5.1 found the hammer, defeated Jr. Troopa, and left the playground, then continued combat and recovery attempts without reaching another story milestone. Astra reached Koopa Bros. Fortress and lowered the first staircase. Neither finished Chapter 1. Cost is why these marathons have a six-hour cap, even when the results make me want to see what another few hours would bring.
Fable 5.1 wasn't the strongest Claude marathon result. An earlier six-hour Opus 5 run at medium effort reached Toad Town, farther than Fable 5.1 managed here. These are individual runs under different harness versions, but they show why I want to test what newer models actually do rather than assume each release will improve on the last.
The marathon timelines and cost breakdown show where the runs diverged and what their recorded tokens would cost at current API rates. The estimates are about $193 for Astra and $68.05 for Fable 5.1. Astra’s estimate prices all non-cached input as cache writes; both estimates exclude unreported usage. Fable’s estimate is lower, while Astra progressed much farther. There is only one six-hour run per model here, so these are runs worth studying rather than settled estimates of consistency.
The story on hsadler.com covers the emulator development, including the failures and breakthroughs that led here.
What I'd improve next
Collecting these results exposed limitations in the setup and the data. We went back through the evidence, corrected errors, and repeated runs, but there are still things I'd want to improve:
- Cost was the biggest reason for using three runs per configuration, even though that's too few to confidently judge consistency. Selecting the latest three valid attempts also pushed older valid results out of the comparison as newer runs arrived. I'd like more repetitions and a clearer collection plan.
- Harness tools, binary versions, and usage records differed, and fixes arrived during collection. I'd like to document every CLI version and use a pinned version within each comparison set, with more complete records from the start.
- I'm still thinking about what makes a fair budget. Some models can spend hours repeating unsuccessful actions, so we need a stopping point. Under the current 90-minute wall-clock cap, faster generation lets a model produce more tokens before time runs out, which makes provider serving speed part of the result. We can't always choose the speed a model is available at. Token, agent-turn, or dollar limits would measure different things, and tokens aren't equivalent across models either. Better timing and usage records would help us understand those tradeoffs before choosing how to compare future runs.
- I'd like to open-source the benchmark and its tools. The emulator fork is already public, but the eval setup needs cleanup, including separating its hi-res dependencies. Knowing something is possible can be enough to start building your own version with agents. That's part of why I want to share how this works before the tooling is ready.
- The opening is already partly saturated: some models finish it consistently. Fixed savestates could put agents directly into combat or puzzles, and other games could test different skills. We could also revisit this route with less help: desktop computer use alone, screenshot-and-controller play, or TAS without walkthroughs and decompiled source. Varying internet access, guidance, and subagents would help show which advantages matter.
These are directions I want to investigate as time and access allow. I don't have a release timeline or a commitment to keep every model up to date.