MODULE m5 · 8.0 HOURS

Faithfulness Evaluation

0% complete

LEARNING OBJECTIVES

Module objectives

Understand the concept of Faithfulness and identify its importance in RAG systems.
Quantitatively evaluate whether generated answers are based on the retrieved context using the Ragas framework.
Analyze the risk of hallucination using automated evaluation metrics.

Faithfulness Evaluation

The core of a Retrieval-Augmented Generation (RAG) system is that an LLM generates answers by utilizing information retrieved from an external knowledge database. Faithfulness is a metric indicating whether the generated answer faithfully reflects only the information described in the retrieved context [S3].

1. Why evaluate Faithfulness?

LLMs tend to answer based on pre-trained knowledge, which can lead to generating information irrelevant to the retrieved context or distorting the context. This is called ‘hallucination’, and faithfulness evaluation can measure this quantitatively [S4].

2. Evaluation Framework: Ragas

Ragas proposes a framework where evaluation is possible without references (reference-free) even when user annotations are absent [S3]. The faithfulness evaluation process generally follows these steps:

Statement Extraction from Answer: Separate verifiable factual statements from the answer.
Evidence Retrieval: Confirm which part of the retrieved context each statement was derived from.
Verification: Judge whether the extracted statements match the context information.

Ragas automates this process by using an LLM as the evaluator [S4].

WORKED EXAMPLES

Worked examples

Example 1: If there is context 'Apples are rich in Vitamin C' and answer 'Apples are rich in Vitamin C, so they are good for immunity', 'good for immunity' is information not in the context, so the faithfulness score decreases.
Example 2: If context 'The foundation date of our company is 2020 year 5 month 1 day' and answer 'The company was founded in 2020 year 5 month' match information, it has a high faithfulness score.

LAB PROTOCOL

Measuring Generated Answer Faithfulness using Ragas

1
Prepare an evaluation dataset (questions, retrieved context, generated answers).
2
Install the Ragas library and import the `Faithfulness` metric.
3
Convert the prepared dataset into Ragas's data structure.
4
Construct an LLM-based evaluator to calculate the faithfulness score of the dataset.
5
Sample answers with low scores and analyze the differences with the retrieved context through human review.

Safety check

Do not send datasets containing private documents or personal information to external LLM APIs.
Check API request rate limits and use caching to control costs.
Maintain security of sample data during human review.

Lab deliverables

Report on the average faithfulness scores of the entire dataset
Dataset analyzing responses with low scores
Comparative analysis between automated evaluation results and human review results

ASSIGNMENT

RAG Pipeline Reliability Evaluation Report

Deliverables

Jupyter Notebook including faithfulness evaluationReport analyzing error classification and frequency of hallucinations

Rubric

Has the faithfulness metric been implemented correctly?
Were hallucination cases in generated responses classified accurately?
Has qualitative consistency between automated evaluation results and human review been secured?

KNOWLEDGE CHECK

Knowledge check

1What is faithfulness in a RAG system?The degree to which retrieved context is highly relevant to the questionThe degree to which generated responses are based on the information in the retrieved contextThe degree to which an LLM utilizes its pre-trained knowledgeThe degree to which the response accurately matches the user's question

2Which of the following is a correct characteristic of the Ragas framework?Evaluation is only possible if human annotations (ground truth) are provided.It supports reference-free evaluation methods.It only evaluates search efficiency and does not evaluate generation quality.It does not use LLMs as evaluators and only uses statistical methods.

FIELD CHECK

Completion criteria

Can quantitatively measure the faithfulness of generated responses using the Ragas library.
Can classify at least 3 types of hallucination occurrence patterns from evaluation results.
Can verify the consistency between the automated evaluation pipeline results and actual responses.

MODULE SOURCES

Module sources