NES Eval
An AI coding agent gets an empty Rust project, the NES hardware documentation and a black-box reference emulator, and has 24 hours to write an NES emulator that compiles to WebAssembly. The emulator is then run in lockstep with the reference on 23 test ROMs and 6 recorded homebrew playthroughs, and every one of 44,980 frames of picture and sound is compared.
It is a port of Mechanize's GBA Eval to a different console. The grader and metrics are theirs; the reference (RustyNES), the ABI, the corpus and the agent environment are new, and so are the problems the NES raised. How the port works · Did any model hack the grader? · Run the emulators in your browser
Leaderboard
| # | Model | Time | Overall | Procedural | Replay | Audio |
|---|---|---|---|---|---|---|
| 1 | GPT-6.1 Sol codex 0.162.1 · high thinking | 24 h | 0.9860 | 0.9300 | 1.0000 | 1.0000 |
| 2 | DeepSeek V4.1 Flash omp 18.11.0 · max thinking | 24 h | 0.9272 | 0.7700 | 0.9955 | 0.8797 |
| 3 | Space Bunny Free opencode 1.18.35 · medium thinking | 2 h | 0.0594 | 0.0000 | 0.0098 | 0.2673 |
| — | Qwen3.8 27B omp 18.10.1 · medium thinking | 24 h | — | never compiled | ||
v1.0 results. Each attempt repository holds the model's working directory saved every 15 minutes, the full transcript and every graded checkpoint. The grader is deterministic: rebuilding a model's final commit and grading it reproduces these numbers to the last digit.
Reference points
| Candidate | Overall | Procedural | Replay | Audio |
|---|---|---|---|---|
| nes-core baseline 0.9426 once the first-call convention is matched; why | 0.5811 | 0.8300 | 0.4569 | 0.7050 |
| RustyNES against itself | 1.0000 | 1.0000 | 1.0000 | 1.0000 |
| Reference + no triangle/noise channel | 0.9452 | 1.0000 | 1.0000 | 0.7261 |
| Reference + A and B swapped | 0.5389 | 1.0000 | 0.3282 | 0.7100 |
| Reference + picture one scanline late | 0.3257 | 0.0000 | 0.2095 | 1.0000 |
| Reference + all three bugs | 0.1948 | 0.0000 | 0.1500 | 0.5241 |
| Blank screen | 0.0039 | 0.0000 | 0.0065 | 0.0000 |
How it is graded
overall = 0.60 × replay + 0.20 × procedural + 0.20 × audio, GBA Eval's weights.
Replay
Six homebrew games, 5,400–6,000 frames each, driven by recorded button presses. Every frame is compared with block SSIM; one frame of timing drift desynchronises the whole playthrough.
Procedural
23 self-checking test ROMs (blargg, kevtris's nestest, Holy Mapperel) for the CPU, PPU, APU and mappers, scored on their final verdict screen.
Audio
Per-frame log-mel spectral distance against the reference's PCM, on the four APU mixer tests and all six games.
Candidates hand over 9-bit palette indices rather than RGB, so a different colour palette cannot cost points, and the power-on state is pinned in the ABI so a correct emulator can match the reference exactly. Full rubric: README; per-testcase scores: Results.
Blog
- 2026-10-16Porting GBA Eval to the NES
What changed when the method moved to a different console: palette indices instead of colours, a power-on state that has to be specified, determinism from the reference cache to the JSON output, and the grader bugs I found before any model ran.
- 2026-10-16Did any model hack the grader?
Four models, 24 hours each. No model special-cased a test ROM, but one constant fitted to the reference is worth 0.36 of a score, one model worked out which emulator it was graded against, and the harness itself had a bug that ended a run early.