Synth Evals · Rust Craftax
Craftax
Model performance across Full, Classic, and multi-agent Craftax.
980Full rollouts
760Classic rollouts
817MA rollouts
3 environments20 shared seeds
Overall
| Model | Composite | Full · historical | Classic | MA | Complete cohorts | Mean suite cost |
|---|---|---|---|---|---|---|
| GPT 6 Luna xhigh | 35.3 | 15.45 /6620 seeds | 16.35 /2220 seeds | 5.30 /6620 seeds | 3 / 3 | $0.2747 |
| DeepSeek V4.1 Flash low | incomplete | 14.65 /6620 seeds | — | 2.10 /6620 seeds | 2 / 3 | — |
| Beam 1.0 max | incomplete | 14.10 /6620 seeds | — | 3.20 /6620 seeds | 2 / 3 | — |
| DeepSeek V4 Flash 0731 low | incomplete | 9.60 /6620 seeds | — | 1.35 /6620 seeds | 2 / 3 | — |
| Nemotron 3.5 Super VL enabled | incomplete | 9.20 /6620 seeds | — | 2.40 /6620 seeds | 2 / 3 | — |
| GPT-4.1 mini none | incomplete | — | 2.95 /2220 seeds | 3.25 /6620 seeds | 2 / 3 | — |
| GPT-4o mini none | incomplete | — | 2.10 /2220 seeds | 1.55 /6620 seeds | 2 / 3 | — |
| GPT-5.4 nano none | incomplete | — | 3.40 /2220 seeds | 0.65 /6620 seeds | 2 / 3 | — |
| GPT-5.6 Luna none | incomplete | — | 3.80 /2220 seeds | 4.05 /6620 seeds | 2 / 3 | — |
| GPT-5 nano minimal | incomplete | — | 0.85 /2220 seeds | 0.30 /6620 seeds | 2 / 3 | — |
| GPT 5.6 Luna high | incomplete | 14.70 /6620 seeds | — | — | 1 / 3 | — |
| MiniMax M3.1 Flash high | incomplete | 14.10 /6620 seeds | — | — | 1 / 3 | — |
| GLM 5.3 Flash high | incomplete | 10.45 /6620 seeds | — | — | 1 / 3 | — |
| Laguna S 2.1 none | incomplete | 5.55 /6620 seeds | — | — | 1 / 3 | — |
| Muse Spark Open low | incomplete | 5.30 /6620 seeds | — | — | 1 / 3 | — |
| GPT 5.4 Nano high | incomplete | 4.90 /6620 seeds | — | — | 1 / 3 | — |
| Laguna XS 2.1 enabled | incomplete | 4.20 /6620 seeds | — | — | 1 / 3 | — |
| Nemotron 3 Super none | incomplete | 3.60 /6620 seeds | — | — | 1 / 3 | — |
| Nemotron 3 Nano none | incomplete | 2.70 /6620 seeds | — | — | 1 / 3 | — |
| Qwen3.6 35B-A3B none | incomplete | — | 2.90 /2220 seeds | — | 1 / 3 | — |
| Qwen3 30B-A3B none | incomplete | — | 2.55 /2220 seeds | — | 1 / 3 | — |
| Qwen3.5 35B-A3B none | incomplete | — | 3.25 /2220 seeds | — | 1 / 3 | — |
| Gemma 3 12B none | incomplete | — | 1.85 /2220 seeds | — | 1 / 3 | — |
| Gemma 4 26B-A4B none | incomplete | — | 3.55 /2220 seeds | — | 1 / 3 | — |
| GPT OSS 120B low | incomplete | — | 4.30 /2220 seeds | — | 1 / 3 | — |
| GPT OSS 20B low | incomplete | — | 4.10 /2220 seeds | — | 1 / 3 | — |
| Llama 3.1 70B none | incomplete | — | 1.50 /2220 seeds | — | 1 / 3 | — |
| Llama 3.1 8B none | incomplete | — | 1.10 /2220 seeds | — | 1 / 3 | — |
| MiniMax M3.1 Flash low | incomplete | — | — | 2.45 /6620 seeds | 1 / 3 | — |
| Nemotron 3.5 Lightning enabled | incomplete | 5.00 /6610 seeds · partial | — | — | 0 / 3 | — |