SWE-bench Verified Is Broken: 5 Things I Found in the Source Code
After building 1,798 SWE-bench containers, I dug into the source. The tests reject correct solutions and every frontier model has memorized the answers.
15 items
After building 1,798 SWE-bench containers, I dug into the source. The tests reject correct solutions and every frontier model has memorized the answers.
SWE-bench's pre-built x86 containers run through QEMU emulation on ARM64 hosts like Apple Silicon and AWS Graviton. I built native ARM64 images and measured a 6.3x speedup on the test runner.
AI coding agents lose critical structural understanding of codebases when context compaction occurs. Code graphs provide persistent external memory—representing functions, classes, and dependencies as queryable relationships—so agents can recover context without re-reading files from scratch.
How Supermodel built Uncompact—a tool that maintains a persistent code graph across Claude Code's context compaction events—and the key lessons learned shipping it to production: simplicity over detail, invisibility enables adoption, and layered verification over blind trust.
A look inside Supermodel's real-time code analysis API: the five-stage processing pipeline, multi-language abstraction via a unified node schema, incremental graph updates, and the sub-100ms response time requirement that shaped every design decision.
Route inference across LLM providers. Track cost per request.
Eval framework. Define correct, test against it, get results.
Ship evals before you ship features.
TypeScript SDK for Supermodel. Generate useful graphs of your codebase.
Benchmark runner for Model Context Protocol servers. Paired comparison experiments on SWE-bench.
OpenAPI spec for the Supermodel public API. Use as reference or generate your own clients.
Supermodel MCP server. Generate code graphs in Cursor, Codex, or Claude Code.
GitHub Action to generate architecture documentation for any repository using Supermodel.
GitHub Action to find unreachable functions using Supermodel call graphs.
Shared core for the MIST stack. Zero external deps.