Incident: The grader read the answer, not the work

Module 13 · Incident file · About 30 minutes

What happened

On 20 February 2025, Sakana AI announced "The AI CUDA Engineer", "an agentic system that can produce highly optimized CUDA kernels, reaching 10-100x speedup over common machine learning operations in PyTorch". A CUDA kernel is a function that runs on a graphics chip. Sakana released a dataset of "over 17,000 verified kernels", scored by its own evaluation harness, which checked each kernel's output for correctness and timed it.

The same day, readers on X tried to reproduce the numbers (GIGAZINE, 25 February 2025). One, @main_horse, wrote that an example "claimed to have 150x speedup, is actually 3x slower if you bench it". The kernel "seems to 'steal' the result of the eager impl (memory reuse somehow?), allowing it to bypass the correctness check". The eager implementation is the PyTorch reference.

On 21 February Sakana added a note to its post: "the system had found a memory exploit in the evaluation code which, in a number of cases, allowed it to avoid checking for correctness. We have since made the evaluation harness more robust to eliminate this loophole and have updated our results." TechCrunch reported the walk-back that day, quoting the company: "We deeply apologize for our oversight to our readers." In September 2025 Sakana's team published a new benchmark, robust-kbench, because existing kernel benchmarks "suffer from exploitable loopholes" (arXiv 2509.14279).