DeepSeek V4.1 Flash
Replays real games almost perfectly (four of six score 0.9998 or better) but fails five test ROMs outright: the four Holy Mapperel board tests and blargg's official-opcode suite. One constant in its frame loop, fitted to the reference, is worth 0.36 of its score; see the post on reward hacking.
| Model | DeepSeek V4.1 Flash |
|---|---|
| Harness | omp 18.11.0 |
| Thinking | max |
| Time limit | 24 hours |
| Environment | NES Eval 77c709b |
| Code | 7,329 lines of Rust; 92 commits; 59 Python tools of its own |
| Tool calls | 3,422; 190 shell commands invoking the oracle directly |
| Tokens | 27.7M input, 4.25M output (3.12M reasoning), 1.59B cache reads; about $9 |
| Agent sessions | 72 (omp -p ends each turn; median 8.8 minutes) |
| Source | github.com/IlyaasK/nes-eval-attempt-deepseek-v4.1-flash |
| Play | run it next to the reference in your browser |
Score
| Checkpoint | Overall | Procedural | Replay | Audio |
|---|---|---|---|---|
| 4h00m | 0.5564 | 0.2200 | 0.6027 | 0.7543 |
| 8h00m | 0.8880 | 0.7000 | 0.9955 | 0.7536 |
| 12h00m | 0.8934 | 0.7000 | 0.9955 | 0.7804 |
| 16h00m | 0.9132 | 0.7000 | 0.9955 | 0.8794 |
| 20h00m | 0.9272 | 0.7700 | 0.9955 | 0.8799 |
| final | 0.9272 | 0.7700 | 0.9955 | 0.8797 |
Replays
The strip under each game is the per-frame penalty: empty means the frame matched the reference, a full-height mark means it scored zero.
Falling 0.9998
46 of 5,800 frames differ, the first at frame 3 · audio 0.9206
Thwaite 0.9894
73 of 6,000 frames differ, the first at frame 4 · audio 0.9120
Double Action Blaster Guys 1.0000
167 of 5,400 frames differ, the first at frame 2 · audio 0.9988
Minekart Madness 1.0000
83 of 5,400 frames differ, the first at frame 2 · audio 0.9853
Nova the Squirrel 0.9835
4,973 of 5,400 frames differ, the first at frame 16 · audio 0.9952
Concentration Room 1.0000
15 of 6,000 frames differ, the first at frame 2 · audio 0.9999
Procedural
| Test ROM | Subsystem | Score |
|---|---|---|
nestest | cpu | 1.0000 |
instr-official | cpu | 0.0000 |
cpu-timing | cpu | 1.0000 |
instr-misc | cpu | 1.0000 |
nmi-and-brk | cpu | 1.0000 |
vbl-basics | ppu | 1.0000 |
nmi-timing | ppu | 1.0000 |
sprite0-basics | ppu | 1.0000 |
sprite0-timing | ppu | 1.0000 |
sprite-overflow | ppu | 1.0000 |
oam-read | ppu | 1.0000 |
apu-len-ctr | apu | 1.0000 |
apu-irq-flag | apu | 1.0000 |
apu-dmc-basics | apu | 1.0000 |
mmc3-clocking | mapper | 1.0000 |
holy-mapperel-mmc1 | mapper | 0.0000 |
holy-mapperel-uxrom | mapper | 0.0000 |
holy-mapperel-cnrom | mapper | 0.0000 |
holy-mapperel-mmc3 | mapper | 0.0000 |
apu-mixer-square | apu | 1.0000 |
apu-mixer-triangle | apu | 1.0000 |
apu-mixer-noise | apu | 1.0000 |
apu-mixer-dmc | apu | 1.0000 |
Failed verdict screens
Reference on the left, this candidate on the right.
instr-official blargg instr_test-v5 official_only: results of all 151 official opcodes, including flags and addressing modes.
holy-mapperel-mmc1 Holy Mapperel MMC1 128K PRG / 8K CHR-RAM: banking and mirroring report.
holy-mapperel-uxrom Holy Mapperel UxROM 128K: banking and mirroring report.
holy-mapperel-cnrom Holy Mapperel CNROM 32K/32K: CHR banking report.
holy-mapperel-mmc3 Holy Mapperel MMC3 128K PRG / 8K CHR-RAM: banking and mirroring report.
