MODULE m8 · 10.0 HOURS
Failure Type Classification and Error Analysis
LEARNING OBJECTIVES
Module objectives
- Able to identify and classify failure types occurring in RAG systems.
- Able to distinguish and analyze errors in the retrieval stage and the generation stage.
- Able to link automated evaluation and human review results by utilizing Ragas framework metrics.
- Able to derive RAG pipeline performance improvement plans based on error analysis data.
Overview of RAG System Error Analysis
The Retrieval Augmented Generation (RAG) architecture consists of a retrieval module and an LLM-based generation module [S3]. When evaluating system performance, it is important to analyze these two stages separately. Errors are broadly divided into issues in the retrieval stage and issues in the generation stage.
1. Classification of Failure Types
- Retrieval Failure: The case where irrelevant or unfocused context is retrieved [S3].
- Generation Failure: The case where the LLM fails to faithfully utilize the provided context (Faithfulness) or generates answers irrelevant to the question [S3].
2. Complementing Automated Evaluation with Human Review
Reference-free frameworks such as Ragas enable the evaluation of retrieval and generation quality without human annotations (ground truth) [S3]. However, automated evaluation metrics alone find it difficult to capture all subtle hallucinations or complex logical errors in the system. Therefore, high-priority failure samples should be extracted via quantitative automated metrics, and actual causes must be identified by necessarily conducting a Human Review alongside them.
WORKED EXAMPLES
Worked examples
- Example 1: When the retriever fetches a 'Model B specifications' document for the question 'What is the release date of Model A?'. This is classified as a 'Retrieval Failure', and embedding model adjustment or search query optimization can be the solution.
- Example 2: When the retriever fetches the correct document for X for the question 'Explain X', but the LLM answers with information not in the document. This is classified as a 'Generation Failure (lack of Faithfulness)', and the constraint 'Use only the provided context' should be reinforced through prompt engineering.
LAB PROTOCOL
Failure Dataset Collection and Error Analysis
- 1
Save the RAG system's answers and retrieved contexts for at least 50 questions.
- 2
Measure Context Precision and Faithfulness for each item using Ragas.
- 3
Extract the bottom 20% question-answer pairs with low metrics.
- 4
Create a classification table for the extracted samples as one of 'Retrieval Error', 'Generation Error', or 'Logic Error'.
- Never include personal information or private data in evaluation code.
- Monitor API call frequency and costs used during the evaluation process to comply with the budget.
- Prevent information leakage by performing tasks in a local environment during data analysis.
Lab deliverables
- Error analysis CSV file with completed classification
- Jupyter Notebook with visualized retrieval and generation quality metrics
ASSIGNMENT
Writing RAG Error Classification and Improvement Report
Deliverables
Rubric
- Accuracy and validity of failure type classification
- Ability to analyze the correlation between quantitative metrics and human review results
- Logical validity of improvement strategies
KNOWLEDGE CHECK
Knowledge check
FIELD CHECK
Completion criteria
- Submission of error classification table including failure types
- Completion of quantitative analysis of retrieval and generation quality utilizing Ragas metrics
- Drafting and review of a pipeline improvement proposal based on error analysis
MODULE SOURCES