OpenAI has discovered approximately 30% error rates in SWE-Bench Pro, a key benchmark for measuring AI coding skills, and has stopped recommending its use.
Imagine you show up for a crucial math exam, only to find the answer key is wrong or the questions themselves are nonsensical. No matter how hard the student tries, they cannot receive an accurate assessment of their skills. The recent controversy surrounding “coding exams” in the AI industry is exactly like this.
OpenAI has actively recommended “SWE-Bench Pro” as a tool for measuring the software engineering capabilities of AI. However, OpenAI recently determined that approximately 30% of these test tasks are incorrectly structured, making it impossible to guarantee the reliability of the results. Consequently, they announced that it is no longer recommended as an official evaluation standard [Source 3, Source 11, Source 4].
Why does this matter?
The smartphone apps, banking systems, and news services we use every day are all powered by code written by developers. Therefore, how well an AI can code is a critical metric that determines how smart the technology we encounter in our daily lives will become.
But what happens if the test questions used to measure this “coding skill” are useless? AI companies have been using these test scores to prove their models are superior. If the test itself is flawed, it could lead to distorted results, such as inflated performance claims or unfair assessments. This poses a significant risk of misleading consumers about the actual level of progress in the AI technology they use.
Easy to understand: The secret of AI coding tests
Coding tests usually calculate scores through “Unit Tests” (automated checks that confirm whether the code correctly performs specific functions). For example, if the test asks the AI to “make it so that pressing this button changes the screen,” the test is processed as correct if the AI writes the code and the test passes.
To use an analogy, imagine hosting a cooking competition where the judges decide to measure “soup thickness,” only to realize their criteria is based on a broken thermometer. No matter how excellent the dish created by the chef (the AI), it’s no different from measuring the skill as low due to a faulty thermometer, or conversely, giving high marks to a dish that shouldn’t have been judged that way.
OpenAI previously used an evaluation tool called “SWE-Bench Verified.” However, this was also discontinued after it was revealed that, along with data contamination issues, approximately 59% of the test questions had structural defects [Source 2, Source 8, Source 13, Source 9].
SWE-Bench Pro, for which recommendations have now been ceased, has also faced criticism that because it creates problems based on complex real-world issues from GitHub (a platform where developers share code), the nature of the tasks is too ambiguous and fragmented for an AI to solve on its own [Source 3, Source 14].
Current situation: Finding a needle in a muddy field
AI models are currently advancing by leaps and bounds. However, the metrics used to measure their performance have not yet achieved the maturity suited for the “AI era” [Source 6, Source 14].
SWE-Bench Pro was released by a company called Scale AI in September 2025, and they attempted to minimize data contamination by applying stronger copyright licenses than before [Source 7, Source 8]. However, this announcement reveals how difficult it is to perfectly translate the complexity of a real-world development environment into an automated test, no matter how meticulously one tries to design it.
Bing Liu, research lead at Scale AI, noted that this decision well illustrates the limitations of relying solely on task ambiguity, data contamination, and narrow unit tests to perfectly measure AI skill [Source 14].
What happens next?
The way we evaluate AI coding skills will undergo fundamental changes.
- More sophisticated evaluation standards: Instead of simply relying on automated scores, the industry will strive to create new standards that can more comprehensively judge the “process” and “complexity” of how AI solves problems [Source 12].
- Emphasis on collaboration with developers: Beyond just AI writing code alone, the method of evaluating how actual developers communicate with AI to solve problems will become more important.
- Continuous verification: Managing evaluation data itself to prevent contamination will become a “basic science” area as important as developing the AI model itself.
We are now living in an era where AI writes its own code. However, this incident reminds us that creating the “thermometer” to accurately evaluate that capability is just as important a challenge as the technological advancement itself.
References
- OpenAI Abandons SWE-Bench Verified, Citing Widespread Data Contamination and Flawed Tests
- OpenAI Retracts Recommendation To Use SWE Bench Pro As Coding Eval Over 30% Broken Tasks
- OpenAI no longer recommends SWE-Bench Pro as coding benchmarks saturate
- Why we no longer evaluate SWE-bench Verified - keynews.ai
- OpenAI Drops SWE-Bench Verified: What It Means for AI
- OpenAI Abandons SWE-bench Verified: 59% Flawed Tests
- OpenAI Drops SWE-bench Verified Over Contamination Concerns
-
[OpenAI Retracts SWE-Bench Pro After Finding 30% of Tasks Broken AlphaSignal](https://alphasignal.ai/news/openai-retracts-swe-bench-pro-after-finding-30-of-tasks-broken) - OpenAI Developers on X
- OpenAI Abandons SWE-bench Verified After Finding 59% of Failed Tests Were Flawed
- OpenAI moves beyond SWE-bench Verified as coding benchmarks saturate
- Because the AI's skills are too good
- Because about 30% of the public tasks are incorrectly structured
- Because all AIs are already achieving perfect scores
- Because the usage fee was too expensive
- Because there were not enough developers
- Due to data contamination and inherent flaws in the test questions themselves
- Scale AI
- Microsoft