MODULE m7 · 10.0 HOURS

Applying Automated Evaluation Framework (Ragas)

0% complete

LEARNING OBJECTIVES

Module objectives

Understand the core evaluation dimensions (retrieval and generation quality) of a RAG pipeline.
Learn how to automatically evaluate RAG performance without references (reference-free) using the Ragas framework.
Analyze and mitigate hallucination risks through quantitative metrics.

Challenges of RAG Evaluation and Ragas

RAG systems consist of a retrieval module and an LLM-based generation module [S3, S4]. Evaluating this structure is a challenging task because it must consider how well the retrieval system identifies relevant context, how faithfully the LLM utilizes the provided context, and what the quality of the response is [S4].

Traditional evaluation methods relied on manual creation and comparison of ground truth, but this is costly and time-consuming, making it unsuitable for fast iteration cycles [S3, S4].

Ragas Framework

Ragas (Retrieval Augmented Generation Assessment) is a framework that can evaluate RAG pipelines even without ground truth datasets [S3, S4]. Ragas automatically evaluates the following core dimensions:

Retrieval Quality: Measures how relevant the retrieved context is to the question (Context Relevance) and whether it contains all necessary information (Context Recall).
Generation Quality: Measures whether the generated response is based on the retrieved context (Faithfulness) and how relevant it is to the question (Answer Relevance).

These metrics utilize LLMs as ‘judges’ to enable evaluation without references, contributing to shortening the RAG development cycle [S3, S4].

WORKED EXAMPLES

Worked examples

Case 1: Context Relevance calculation. Ragas uses an LLM to extract sentences actually needed to answer the question from the retrieved context, and calculates the score through the ratio of needed sentences to the total context.
Case 2: Faithfulness evaluation. An LLM determines whether each claim in the generated response is supported by the retrieved context. The more unsupported claims there are, the higher the hallucination score becomes.

LAB PROTOCOL

Practice Quantitatively Evaluating RAG Performance using Ragas

1
Prepare an evaluation dataset (question, retrieved context, generated response).
2
Install the `ragas` library in a Python environment.
3
Convert the evaluation dataset into a `ragas` Dataset object.
4
Call Ragas' `evaluate` function to calculate metrics such as Context Relevance and Faithfulness.
5
Visualize the result values and analyze queries with low scores.

Safety check

Ensure that personal information or private data is not included in the document corpus used for evaluation.
Actively utilize local models or caching during testing to prevent API usage costs.
Automated evaluation results are supplementary indicators of reliability; cross-validate the confirmation of actual model quality in parallel with sample human review.

Lab deliverables

Evaluation result dataframe containing metric scores
Logs analyzing query types that received low scores

ASSIGNMENT

RAG Pipeline Performance Comparison Report

Deliverables

Ragas evaluation results for two RAG pipelines with different search settings (k-value, embedding model, etc.)Performance analysis report between the two settings

Rubric

Were Ragas metrics (Context Relevance, Faithfulness, etc.) implemented correctly?
Are evaluation results quantitatively compared and interpreted logically?
Were hallucination types classified for at least 3 cases and were improvement plans suggested?

KNOWLEDGE CHECK

Knowledge check

1What is the biggest characteristic of the Ragas framework?A human answer dataset is absolutely requiredIt can evaluate RAG pipelines without references (reference-free)It only evaluates the search stage and does not evaluate the generation stageAt least 10 GPU(s) are absolutely required

2What is the definition of the 'Faithfulness' metric measured by Ragas?How relevant retrieved context is to the questionWhether the question exists within the document corpusWhether the generated response is based on retrieved contextHow much the questioner trusts the LLM's response

FIELD CHECK

Completion criteria

Successfully calculate at least 4 metrics for at least 10 queries using the Ragas library.
The lab notebook is regularly committed to the Git repository.
The performance comparison report includes at least 3 error classification cases.

MODULE SOURCES

Module sources