Results
Every graded testcase for every candidate that produced a binary. Qwen3.8 27B is absent because none of its commits compiled. Hover a testcase for what it checks.
| Section | GPT-6.1 Sol | DeepSeek V4.1 Flash | Space Bunny Free | nes-core |
|---|---|---|---|---|
| procedural | 0.930 | 0.770 | 0.000 | 0.830 |
| replay | 1.000 | 0.995 | 0.010 | 0.457 |
| audio | 1.000 | 0.880 | 0.267 | 0.705 |
| overall | 0.986 | 0.927 | 0.059 | 0.581 |
Replay 60% · video score, mean over every frame
| Testcase | GPT-6.1 Sol | DeepSeek V4.1 Flash | Space Bunny Free | nes-core |
|---|---|---|---|---|
Fallinggameplay · 5,800 frames | 1.000 | 1.000 | 0.006 | 0.283 |
Thwaitegameplay · 6,000 frames | 1.000 | 0.989 | 0.001 | 0.310 |
Double Action Blaster Guysgameplay · 5,400 frames | 1.000 | 1.000 | 0.020 | 0.038 |
Minekart Madnessgameplay · 5,400 frames | 1.000 | 1.000 | 0.001 | 0.997 |
Nova the Squirrelgameplay · 5,400 frames | 1.000 | 0.983 | 0.031 | 0.999 |
Concentration Roomgameplay · 6,000 frames | 1.000 | 1.000 | 0.000 | 0.114 |
Procedural 20% · video score of the final verdict screen
| Testcase | GPT-6.1 Sol | DeepSeek V4.1 Flash | Space Bunny Free | nes-core |
|---|---|---|---|---|
nestestcpu · 120 frames | 1.000 | 1.000 | 0.000 | 1.000 |
instr-officialcpu · 2,430 frames | 1.000 | 0.000 | 0.000 | 1.000 |
cpu-timingcpu · 840 frames | 1.000 | 1.000 | 0.000 | 1.000 |
instr-misccpu · 360 frames | 1.000 | 1.000 | 0.000 | 1.000 |
nmi-and-brkcpu · 210 frames | 0.000 | 1.000 | 0.000 | 1.000 |
vbl-basicsppu · 240 frames | 1.000 | 1.000 | 0.000 | 1.000 |
nmi-timingppu · 330 frames | 1.000 | 1.000 | 0.000 | 0.000 |
sprite0-basicsppu · 120 frames | 1.000 | 1.000 | 0.000 | 1.000 |
sprite0-timingppu · 150 frames | 1.000 | 1.000 | 0.000 | 1.000 |
sprite-overflowppu · 120 frames | 1.000 | 1.000 | 0.000 | 1.000 |
oam-readppu · 120 frames | 1.000 | 1.000 | 0.000 | 1.000 |
apu-len-ctrapu · 120 frames | 1.000 | 1.000 | 0.000 | 1.000 |
apu-irq-flagapu · 120 frames | 1.000 | 1.000 | 0.000 | 1.000 |
apu-dmc-basicsapu · 120 frames | 1.000 | 1.000 | 0.000 | 1.000 |
mmc3-clockingmapper · 120 frames | 1.000 | 1.000 | 0.000 | 0.000 |
holy-mapperel-mmc1mapper · 270 frames | 1.000 | 0.000 | 0.000 | 0.000 |
holy-mapperel-uxrommapper · 180 frames | 1.000 | 0.000 | 0.000 | 1.000 |
holy-mapperel-cnrommapper · 120 frames | 1.000 | 0.000 | 0.000 | 1.000 |
holy-mapperel-mmc3mapper · 270 frames | 1.000 | 0.000 | 0.000 | 0.000 |
apu-mixer-squareapu · 1,290 frames | 1.000 | 1.000 | 0.000 | 1.000 |
apu-mixer-triangleapu · 840 frames | 1.000 | 1.000 | 0.000 | 1.000 |
apu-mixer-noiseapu · 1,530 frames | 1.000 | 1.000 | 0.000 | 1.000 |
apu-mixer-dmcapu · 960 frames | 1.000 | 1.000 | 0.000 | 1.000 |
Audio 20% · log-mel spectral score
| Testcase | GPT-6.1 Sol | DeepSeek V4.1 Flash | Space Bunny Free | nes-core |
|---|---|---|---|---|
apu-mixer-squareapu · 1,290 frames | 1.000 | 0.760 | 0.191 | 0.961 |
apu-mixer-triangleapu · 840 frames | 1.000 | 0.975 | 0.197 | 0.998 |
apu-mixer-noiseapu · 1,530 frames | 1.000 | 0.588 | 0.391 | 0.553 |
apu-mixer-dmcapu · 960 frames | 1.000 | 0.840 | 0.204 | 0.979 |
Fallinggameplay · 5,800 frames | 1.000 | 0.921 | 0.023 | 0.523 |
Thwaitegameplay · 6,000 frames | 1.000 | 0.912 | 0.066 | 0.296 |
Double Action Blaster Guysgameplay · 5,400 frames | 1.000 | 0.999 | 0.594 | 0.601 |
Minekart Madnessgameplay · 5,400 frames | 1.000 | 0.985 | 0.189 | 0.797 |
Nova the Squirrelgameplay · 5,400 frames | 1.000 | 0.995 | 0.706 | 0.835 |
Concentration Roomgameplay · 6,000 frames | 1.000 | 1.000 | 0.156 | 0.172 |