← Blog

Porting GBA Eval to the NES

Ilyaas Kapadia · 2026-10-16

GBA Eval gives a frontier coding agent 24 hours to write a Game Boy Advance emulator from scratch, then grades it by replaying recorded gameplay in lockstep with a reference emulator and comparing every frame. The idea it rests on is simple: a console has no clock, no network and no source of randomness, so the same game and the same button presses always produce the same frames. That makes "is this emulator correct?" a question you can answer automatically, frame by frame, without a human looking at the screen.

NES Eval applies the same method to the NES. I reused what could be reused: the lockstep grader, the video and audio metrics, the section weights and the shape of the agent environment are all ported from GBA Eval's open-source harness. This post is about the parts that didn't port: the decisions the NES forced, how I made the whole pipeline deterministic, and the problems I found in the grader before any model touched it.

The task#

The agent runs in a container with an empty Rust crate, a task description, an ABI to implement, an offline copy of the relevant NESdev wiki pages, and a set of development ROMs: four homebrew games and 30 test ROMs, none of which are graded. It also gets an oracle command. The reference emulator runs in a separate sidecar container, and the agent can ask it to run any ROM with any inputs and dump frames, audio or session state. It never sees the reference's code, or even its name.

The deliverable is a wasm32-unknown-unknown module exporting ten C functions: load a ROM, set the controller, run one frame, read the framebuffer and the audio. The grader loads it in wasmtime with every import trapping and a fuel limit per call, feeds it the same ROM and the same button presses as the reference, and scores:

That's GBA Eval's design, unchanged. Everything below is where the NES needed something different.

Picking a reference#

The reference is the ground truth: a candidate is "correct" exactly when it agrees with it. GBA Eval uses Mesen2, a mature C++ emulator, wrapped behind their ABI. Mesen2 also emulates the NES, and it was my first instinct.

I used RustyNES instead. It is written in Rust, so it compiles to the same wasm32-unknown-unknown target the candidates do, behind a thin shim implementing the same ABI. The grader therefore treats the reference as just another candidate: grader --reference any.wasm grades against any module implementing the ABI, and the reference cache is keyed by that module's hash. Its core reads no clock, spawns no threads and touches no OS randomness, and it passes all of AccuracyCoin, a large suite of cycle-level hardware tests.

The cost is a less battle-tested reference than Mesen2, and a public one. Every replay is graded against RustyNES, not against hardware, so where RustyNES is wrong in a way no test ROM catches, a more accurate candidate loses points. GBA Eval has the same exposure with Mesen2; the mitigation is the same too, which is the next section but one.

Emulators hand over palette numbers, not colours#

The GBA outputs 15-bit RGB, and GBA Eval has to cope with emulators expanding 5-bit channels to 8 bits in different ways, so it quantises back to 5 bits before comparing. The NES has a different problem: it doesn't output RGB at all. The PPU produces a 6-bit palette number plus three "emphasis" bits per pixel, and the colour you see depends on how a composite video signal is decoded. Every emulator ships its own palette, and none of them is "correct".

If candidates returned RGB, an emulator with perfect timing and a different palette table would lose points on every pixel of every frame. That measures taste, not accuracy. So the ABI asks for the 9-bit value (emphasis << 6) | colour per pixel, and the grader maps it to RGB with one fixed table before computing similarity. Two pixels now match exactly when two emulators computed the same thing, which also made GBA Eval's quantisation step unnecessary, and the reference cache stores what the NES computed rather than what one emulator chose to display.

The cost: a candidate with a wrong RGB table is invisible to the benchmark. That's intended, and the README says so.

One smaller change in the same place: GBA Eval's video metric computes SSIM over 10×10 blocks. The NES frame is 256×240, which 10×10 doesn't divide, and would leave a 6-pixel column unscored. 8×8 divides it exactly into 960 blocks that also line up with the NES's tiles.

Making it deterministic, end to end#

The whole method depends on one property: the same inputs produce the same outputs, forever. If the reference isn't deterministic under the harness, nothing else matters, because the cache it produces isn't ground truth. So the first thing I built was a probe that ran the reference twice per ROM and compared the bytes. Then I went looking for every other place non-determinism could get in. There turned out to be four.

1. The console itself

The NES is not quite as deterministic as the GBA. Work RAM powers up with unpredictable contents, the CPU and PPU clocks start at a random alignment relative to each other, and some games seed their random number generators from exactly those things. On hardware, two power-ons of the same cartridge can play differently.

An emulator has to pick something, and if the spec says nothing, a candidate can be correct to the hardware and still disagree with the reference from frame zero. So the power-on state is part of the ABI: zeroed RAM, the CPU registers at the first instruction (P=$24, S=$FD), the PPU at scanline 0, dot 25 at that moment, and a fixed clock alignment. I got these by probing the reference rather than by trusting its documentation. My first draft of the spec said P=$34 and dot 0, and both were wrong.

The baseline emulator I wrote (more on it below) found more gaps. It gave cartridges exactly the PRG-RAM their NES 2.0 header declares, which for two mapper tests is none, while the reference always gives MMC1 and MMC3 boards 8 KiB; it scored 0 on both. It also diverged from every game within the first 16 frames. I guessed the cause was the PPU warm-up: the reference ignores PPU register writes for the first 29,658 CPU cycles after power-on, as the NES does, and the baseline doesn't. Both are behaviours a reasonable emulator could get "wrong" relative to the reference, so both went into the spec.

The warm-up belongs in the spec, but it wasn't the cause of the divergence. The real cause was a third gap, which a model run exposed and which turned out to be worth 0.36 of a score; that's in the next post. The lesson holds either way: an underspecified ABI looks fine until an independent implementation hits it.

2. WebAssembly

Wasm is deterministic with one exception: the bit pattern of a NaN produced by floating-point arithmetic is allowed to differ between machines. An emulator that does audio filtering in floating point can, in principle, produce different bytes on a different CPU. The grader runs wasmtime with NaN canonicalisation on, so a reference cache built on one machine matches a candidate graded on another.

3. The harness

The ported grader had two sources of non-determinism of its own. Its summary included a wall-clock timestamp, which I removed. And it aggregated scores through hash maps, so floating-point sums happened in a different order on each run, and the last digit of a score could change between two gradings of the same file. Every map is now ordered. Testcases still run in parallel, but each has its own state and results are combined in manifest order.

4. The build

The reference cache is keyed by the reference wasm's SHA-256, so the wasm must build to the same bytes every time. The toolchain is pinned, and one surprise is documented: the reference wasm's hash depends on its source line numbers, because panic messages embed them. Any edit to the shim, even a comment, invalidates every cache.

The gate

scripts/determinism-gate.sh checks all of this at once. It regenerates every reference cache from scratch twice and requires byte-identical files, then grades a candidate twice and requires every output file to be byte-identical. The last run produced identical caches for all 29 testcases and 146 identical result files. Anyone can rebuild a model's final commit from its attempt repository and get its score to the last digit.

Only grade test ROMs the reference passes#

The procedural section uses self-checking test ROMs (blargg's suites, kevtris's nestest, Holy Mapperel), scored on their final screen. Before accepting each one I rendered the reference's end screen and read it. One, mmc3_test_2/4-scanline_timing, ends with "Failed #9" on RustyNES.

Keeping it would mean a candidate scores higher by reproducing a reference bug than by being right. It's out of the graded set, and it ships with the agent's development ROMs instead, where the agent can see the reference's behaviour but isn't graded on it. The general problem doesn't go away, since every replay is graded against RustyNES, but a reference with a strong test record and a procedural set it demonstrably passes keeps it small.

The audio metric ignores loudness#

To check the grader catches what it should, I built three broken versions of the reference, each with one injected bug, plus one with all three:

Each bug lands where it should. The one-scanline offset fails every verdict screen but leaves audio untouched. Swapped buttons can't affect test ROMs that never press A or B, but games that use them play out differently; Falling, steered with the D-pad only, still scores 1.0. Missing channels cost only audio.

But one number stood out. Without the triangle and noise channels, blargg's apu_mixer/triangle scored 0.961 on audio. That test plays a triangle wave and an inverted copy of it through the DMC channel, so on a correct emulator they cancel to near-silence. Remove the triangle and the DMC copy is left playing, about 10 times louder than the reference. GBA Eval's audio metric subtracts each frame's mean log-mel level before comparing spectra, a deliberate choice so that emulators with different output gain aren't punished. After that subtraction, a loud DMC tone and a quiet residual have the same shape, and the metric calls them a match.

The noise test (0.058) and the game replays (0.39 to 0.92) catch the missing channels, so the bug doesn't go unnoticed overall. It is still a real blind spot: any bug that changes how loud something is without changing its spectrum is graded as correct. The fix I'd propose is a bounded loudness term, for example penalising per-frame RMS ratios outside [0.5, 2], which still tolerates gain differences. I haven't added it, to keep the metric identical to GBA Eval's until a model run shows it matters. The grader already reports audio_rms_ratio per frame, so that's checkable.

A homebrew corpus#

Commercial games are out, for licensing reasons, which also rules out TASVideos movies: nearly all of them are commercial, and recorded on a different emulator. The replays use six homebrew games, each downloaded from its author's own release with its license checked at the source:

I wrote each replay against the reference with a small native probe tool and checked it from screenshots every 120 frames. Each one reaches gameplay within the first 300 frames and covers scrolling, scoring, deaths, level changes and, in Falling, a game over. One game I found was dropped because its music had no license, and Nova the Squirrel's assets are non-commercial only, which the README flags along with how to remove it.

The agent gets four different homebrew games and 30 test ROMs that aren't graded, mirroring GBA Eval's split between development and held-out ROMs.

What a middling score looks like#

GBA Eval ships gba-core, a modest emulator scoring about 0.53, so a reader knows what a mid-table number means. nes-core is the NES equivalent: 2,600 lines I wrote from the NESdev wiki without looking at any other emulator, graded once and not tuned afterwards, because a baseline tuned against the corpus stops being a baseline. It scores 0.5811.

It passes 19 of 23 test ROMs (procedural 0.83) but scores only 0.46 on replays, and the replay breakdown is the interesting part. In two games it stays close to the reference (0.997 and 0.999). In the other four, the baseline is one frame out of step from the first call onwards, so every recorded button press reaches the game at a different moment than when the replay was recorded, and the playthrough diverges for good: 95–97% of frames differ.

That is what replay grading measures. It doesn't ask whether an emulator looks right; it asks whether it behaves identically, and a one-frame difference compounds the way it would derail a tool-assisted speedrun. That sensitivity is what keeps strong models from all scoring 0.99. It is also a risk, and here it bit: the one frame turned out not to be an emulation error at all, but an unwritten convention about the first call. With that one convention matched, the same untuned baseline scores 0.9426. The details, and how a model found it before I did, are in the next post.

What changed, in one table#

Four models have now been through the full pipeline, from an empty task container to a published attempt repository. What they did, including whether any of them tried to game the grader, is the next post.