MODULE m10 · 18.0 HOURS

Reproducible Regression Evaluation Reporting

0% complete

LEARNING OBJECTIVES

Module objectives

  1. Understand the framework for reproducible regression evaluation of RAG systems.
  2. Quantitatively measure retrieval and generation quality using the Ragas framework.
  3. Analyze the impact of model updates or search algorithm changes on system reliability through regression tests.
  4. Learn comprehensive reporting methods through the harmony of human review and automated evaluation.

Key Dimensions of RAG System Evaluation

Evaluating RAG architecture is a multifaceted task. The ability of the search system to identify highly relevant and focused context to questions, the ability of the LLM to faithfully generate answers using the identified context, and the quality of the final generated product itself are subjects of evaluation [S3, S4].

Ragas Framework

Ragas (Retrieval Augmented Generation Assessment) is a framework that can evaluate RAG pipelines without Ground Truth [S3]. Ragas provides a series of metrics to measure Retrieval quality, Generation quality, and Hallucination prevention capabilities [S3].

Importance of Regression Evaluation

Change Management is essential to maintain system reliability. Regression tests must be performed on existing evaluation datasets when introducing new embedding models, tuning search algorithms, or changing LLM settings. A regression evaluation report serves as data statistically proving whether the system’s improvements lead to actual reliability enhancement or induce new defects.

WORKED EXAMPLES

Worked examples

  1. Statistical comparison example: A notebook analysis case that executes the same 100 question evaluation set for two RAG settings (existing vs. new embedding model) and compares the mean and standard deviation of Ragas metrics (Faithfulness, Answer Relevance) to verify significant performance improvement.
  2. Error type classification example: A method of sampling 30 cases where the system resulted in low 'Answer Relevance' scores, manually classifying whether it is a failure in the retrieval stage (failure to retrieve relevant documents) or the generation stage (ignoring context), and recording this in pipeline logs.

LAB PROTOCOL

RAG Pipeline Regression Test Automation

  1. 1

    Prepare a final validation evaluation dataset (100 questions) in JSON format.

  2. 2

    Define two different RAG pipeline settings (Version A, Version B).

  3. 3

    Perform automated evaluation for each pipeline using the Ragas framework and save the results.

  4. 4

    Use Pandas to visualize the distribution of metrics for both result sets and calculate statistical differences.

  5. 5

    Compare the evidence context and model responses for the bottom 10% of cases where evaluation scores dropped sharply.

Safety check
  • Ensure that the evaluation dataset does not contain personal information or internal company confidential documents.
  • Set cost limits when calling external APIs, and use caching when testing in a local environment to prevent excessive or redundant requests.
  • Do not rely solely on model evaluation results; always accompany them with human review (Human-in-the-loop) for a sample of the data.

Lab deliverables

  • Jupyter Notebook containing the results of the regression evaluation.
  • Visualization graph (boxplot or scatter plot) comparing performance between the two RAG configurations.
  • Final report including error type classification and human review records.

ASSIGNMENT

Writing a RAG Reliability Improvement Report

Deliverables

Rubric

  • Have retrieval and generation quality metrics been measured quantitatively?
  • Is the regression test methodology described and reproducible?
  • Is the analysis between automatic evaluation results and human review results appropriate?
  • Are the causes of performance changes and future improvement directions clearly presented?

KNOWLEDGE CHECK

Knowledge check

1What is the most significant feature of the Ragas framework?
2What is the primary purpose of performing regression tests in a RAG system?
3Which of the following is not a multidimensional aspect that should be considered when evaluating a RAG system?

FIELD CHECK

Completion criteria

  • Design a regression test pipeline and complete a comparative analysis of at least 2 configurations using a dataset of at least 100 items.
  • Perform quantitative evaluation using Ragas metrics.
  • Submit records of verifying automatic evaluation results through human sample review.
  • Write and submit a technical report.

MODULE SOURCES

Module sources

  1. [2309.15217] Ragas: Automated Evaluation of Retrieval ... Ragas: Automated Evaluation of Retrieval Augmented Generation RAGAS: Automated Evaluation of Retrieval Augmented Generation RAGAs: Automated Evaluation of Retrieval Augmented Generation RAGAS: Automated Evaluation of Retrieval Augmented Generation RAGAS: Automated Evaluation of Retrieval Augmented Generationarxiv.org · paper
  2. Ragas: Automated Evaluation of Retrieval Augmented Generationarxiv.org · paper