MODULE m6 · 8.0 HOURS

Citation Accuracy and Source Tracking

0% complete

LEARNING OBJECTIVES

Module objectives

  1. Understand how faithfully a response reflects the content of retrieved documents in a RAG system.
  2. Learn the definition and measurement methods of citation accuracy.
  3. Quantitatively evaluate the faithfulness and answer relevance of responses using the Ragas framework.
  4. Design a process to verify the traceability of sources in model responses.

RAG System Citation and Faithfulness Evaluation

RAG (Retrieval Augmented Generation) systems utilize external knowledge bases to reduce the risk of LLM hallucinations, but a process to verify whether generated responses accurately cite retrieved documents is essential [S4].

1. Key Evaluation Metrics

  • Faithfulness: Measures whether generated responses are derived from the provided retrieved context. Every claim in the response must be based on retrieved documents; scores are lower if responses rely solely on external knowledge or the model’s pre-trained knowledge [S4].
  • Answer Relevance: Evaluates how directly relevant the response is to the given question. This is used to identify cases where a response deviates from the intent of the question, even if retrieved information is sufficient.

2. Citation Accuracy Verification Process

Citation accuracy is the process of identifying which part of the retrieved context a specific sentence in the response cites and verifying if it aligns with facts in the original text. The automated evaluation framework, Ragas, provides metrics that can evaluate faithfulness even without reference data (ground truth) for this process [S4].

3. Failure Type Classification

  • Citation Omission: Facts in the response are present in retrieved documents, but no citation is indicated.
  • False Citation: Indicates as if it cited content not found in retrieved documents.
  • Fact Distortion: Citations are correctly indicated, but generated by misinterpreting the meaning of the original text.

WORKED EXAMPLES

Worked examples

  1. Example 1: Faithfulness score calculation. If the question 'What is A's revenue in 2025?' generates the response 'A's revenue in 2025 was 100 billion.', and the context document includes 'A recorded 100 billion in revenue in 2025.', since all information in the response exists within the context, the faithfulness score is evaluated as 1.0 (maximum).
  2. Example 2: Citation accuracy error identification. If the question 'When was A founded?' generates the response 'A was founded in 1990 (Reference: Document 1).', but Document 1 specifies 'A was founded in 1995', this is classified as a 'Fact Distortion' failure type and is judged as having low citation accuracy.

LAB PROTOCOL

Automated Response Faithfulness Evaluation Practice using Ragas

  1. 1

    Prepare a dataset of search results (Context) and generated responses (Answer) for the RAG system to be evaluated.

  2. 2

    Install the Ragas framework and load the response dataset.

  3. 3

    Calculate scores for each question-answer pair in the dataset using Ragas' `Faithfulness` metric.

  4. 4

    Separately extract 30 responses with a faithfulness score of less than 0.7.

  5. 5

    Human-review the extracted samples and tag them with corresponding failure types: 'Citation Omission', 'False Citation', or 'Fact Distortion'.

Safety check
  • Ensure anonymization is completed in advance so that personal information or corporate secrets are not included in the document corpus used during the evaluation process.
  • Set cost limits when calling external APIs and fix seed values for reproducibility to prevent repetitive API costs.

Lab deliverables

  • Jupyter Notebook file (.ipynb) containing evaluation results
  • Visualization chart of faithfulness score distribution
  • Failure type classification table including human review records

ASSIGNMENT

RAG System Reliability Regression Evaluation Report Writing

Deliverables

Rubric

  • Is the quantitative calculation method for evaluation metrics accurately specified?
  • Is the citation relationship between retrieved documents and generated responses logically traceable?
  • Is the failure type classification consistent with human review data and does it provide valid evidence?

KNOWLEDGE CHECK

Knowledge check

1Which explanation is correct regarding the 'Faithfulness' metric in the Ragas framework?
2Which case corresponds to the 'Fact Distortion' failure type when evaluating citation accuracy?

FIELD CHECK

Completion criteria

  • Completion of automated evaluation metric calculation via Jupyter Notebook
  • Submission of human review and failure type classification records for at least 30 response samples
  • Writing a final report containing evaluation results and improvement plans

MODULE SOURCES

Module sources

  1. Ragas: Automated Evaluation of Retrieval Augmented Generationarxiv.org · paper