5 items
How SWE-bench Verified's fail_to_pass and pass_to_pass tests and task instances actually work — and why every frontier model score is contaminated. Source code analysis.
SWE-bench's pre-built x86 containers run through QEMU emulation on ARM64 hosts like Apple Silicon and AWS Graviton. I built native ARM64 images and measured a 6.3x speedup on the test runner.
Eval framework. Define correct, test against it, get results.
Ship evals before you ship features.
Benchmark runner for Model Context Protocol servers. Paired comparison experiments on SWE-bench.