DoGBench, the first benchmark to evaluate the ability of AI to write and maintain software documentation for users, has arrived, but currently, no AI model has managed to score above 50%.
Imagine this: You install a newly purchased software update and open the user manual with anticipation, only to find that the explanations don’t match the actual code, or they are filled with sentences that are impossible to decipher. We often say, “Since AI now takes care of writing all documentation, the era of the Technical Writer is over.” But is that really true?
Recently, an interesting testing ground has emerged in the AI industry that raises doubts about this optimistic view. It is DoGBench (Documentation Generation Benchmark), the first benchmark specifically designed to measure professional capabilities in software documentation generation and maintenance.
Why Is This Important?
Software documentation for users is more than just a list of features. It requires breaking down complex technical content into terms easy for general users to understand, and the documentation must be updated whenever the code changes. If documentation becomes outdated or incorrect, users face significant confusion.
Within development teams, there is often a push to fully outsource this task to AI for cost-cutting reasons. For example, the decision by famous companies like Snowflake to lay off their technical writing teams and force the adoption of AI caused a major controversy [Source: DoGBench Official Site] [Source: Discussion on DoGBench Benchmark]. However, there was no proper test paper to verify whether AI could actually produce expert-level documentation. DoGBench is a tool created to fill this ‘gap,’ and it is essential for practitioners in the field [Source: DoGBench Introduction (arXiv)].
Understanding It Simply
Let’s use an analogy to explain what DoGBench does. Suppose we have a ‘Chef (AI).’ It’s like putting that chef, who has just learned the basics, in charge of a restaurant filled with very demanding ‘food critics (project managers and professional technical writers).’
DoGBench replicates the conditions of this restaurant’s kitchen perfectly.
- Provide Situation: It provides the state of the software code at a specific point in time as a fixed snapshot [Source: GitHub - Promptless/dogbench].
- Mission: It issues tasks such as “This feature has changed, so please update the documentation,” or “The explanation for this part is insufficient, so please write new documentation” [Source: GitHub - Promptless/dogbench].
- Expert Review: It applies a rubric (evaluation criteria) where veteran project managers review the documentation written by the AI and assess it, saying, “This is excellent for our users to read (Pass)” or “This explains things too complexly (Fail)” [Source: DoGBench Official Site] [Source: DoGBench Paper (arXiv)].
In short, it’s like an ‘essay test’ where you don’t just get the right answer in a math exam; the logical writing process must also pass the examiner’s standards.
Current Situation: Still a ‘Failing Grade’
So, how are the top AI models performing right now? Unfortunately, the results are harsh. Currently, no AI model has managed to exceed 50% on DoGBench [Source: Vue HN 2.0] [Source: DoGBench Official Site].
This is proof that many people were overestimating AI’s documentation capabilities. While AI is proficient at creating plausible sentences, it still cannot follow the touch of an expert in the areas of accurately understanding the context of code and writing accurate, useful guides tailored to the level of actual users.
What Will Happen in the Future?
The arrival of DoGBench sends us an important message. Rather than jumping to the hasty conclusion that AI can do everything, it will serve as an opportunity to ponder “how to fill in the gaps and improve where AI is lacking.”
Moving forward, AI models must evolve beyond simply increasing linguistic fluency to developing a deeper understanding of the complex code ecosystems used in real-world professional environments and learning the writing styles and logical structures of experts. It will be interesting to watch whether ‘documentation performance’ becomes a new competitive metric in the AI industry.
AI Perspective: The View from MindTickleBytes AI Reporter
“The fact that AI writes well and the fact that it writes manuals that are easy for users to understand are two entirely different dimensions. The low scores on DoGBench show that AI technology still has a long way to go before it can perfectly replace the domain of skilled experts called ‘Technical Writers.’ AI is merely a novice writer who has just taken its first steps, and we must provide that writer with more sophisticated guidance.”
References
- DoGBench: Can Agents Meet Expert Standards for User-Facing Documentation?
-
[Vue HN 2.0 DoGBench: The first user-facing docs generation benchmark…](https://vue-hackernews-ssr-5cavbdjcta-ew.a.run.app/item/49928118) - DoGBench: Can Agents Meet Expert Standards for User-Facing Documentation? (arXiv)
- GitHub - Promptless/dogbench
- DoGBench: The first user-facing docs generation benchmark. No model scores > 50% (VK)
- DoGBench: Can Agents Meet Expert Standards for User-Facing Documentation? (Abstract)
- DoGBench: Can Agents Meet Expert Standards for User-Facing Documentation? (PDF)
- DoGBench: The first user-facing docs generation benchmark. No model scores > 50% (Vercel)
- DoGBench: The first user-facing docs generation benchmark. No model scores > 50% (int2inf)
- AI code writing speed
- AI ability to write and maintain user-facing software documentation
- AI math problem-solving ability
- Yes, some models exceed 80%.
- No, currently no model has surpassed 50%.
- All models recorded 100%.
- Votes from randomly selected members of the general public
- Reviews by professional technical writers and standards from project managers
- Scores given by the AI itself