MODULE m3 · 8.0 HOURS

Question-Answer Evaluation Set Construction

0% complete

LEARNING OBJECTIVES

Module objectives

Understand the necessity of a high-quality Question-Answer (QA) evaluation set for RAG system evaluation.
Grasp the principles of building evaluation sets utilizing Synthetic Data Generation techniques.
Master the logic for evaluating the factual consistency of model-generated answers through methodologies like TrueTeacher.
Design a process for generating datasets that are robust to domain changes while maintaining evaluation set quality.

Building Question-Answer (QA) Evaluation Sets for RAG

To reliably measure the performance of a Retrieval-Augmented Generation (RAG) system, an elaborately designed evaluation set is essential. Relying solely on human-written questions and answers has limitations in terms of cost and scalability for large-scale evaluation.

The Importance of Synthetic Data Generation

According to the recent TrueTeacher methodology, synthetic training data can be generated by annotating various model-generated answers using an LLM [S5]. This approach has the following advantages:

Cost Efficiency: Since it does not rely on human-written summaries or answers, large-scale dataset generation (e.g., 1.4M examples) is possible [S5].
Multilingualism and Scalability: It is not limited to a specific language and shows robustness against Domain-shift [S5].
Factual Consistency Evaluation: Small models trained using synthetic data can successfully distill the knowledge of large LLM teacher models and be used as efficient factual evaluation tools [S5].

Dataset Construction Strategy

When constructing an evaluation set, it must be composed not only of question-answer pairs but also to measure ‘Does the search result contain the evidence needed to derive the correct answer?’ and ‘Does the model refer to that evidence without distortion?’. To this end, when generating the dataset, the complexity of questions, relevance to search results, and factual consistency of answers must be systematically labeled or verified.

WORKED EXAMPLES

Worked examples

Example 1: Extracting core passages from a document corpus. A pipeline to extract factually significant sentences in context from a given document using an LLM, and based on this, generate 'answerable questions' and 'negative samples'.
Example 2: Designing a factuality verification prompt. A process to refine ground truth for evaluation by instructing an LLM based on generated questions and retrieved documents: 'Answer the question based on the retrieved document, and judge whether the answer is factually consistent with True/False'.

LAB PROTOCOL

Synthetic Evaluation Set Generation Practice

1
Load the prepared open-license document corpus and split it into text chunks.
2
Generate 100 or more unique question-answer pairs extractable from each chunk using the LLM API.
3
Simulate a search system for the generated questions to retrieve the top-k documents.
4
Construct an evaluation pipeline that judges factual consistency between retrieved documents and generated answers.
5
Save the result data in JSONL format, and manually review 30 samples to record data quality.

Safety check

When using external APIs in the evaluation set construction process, you must set an API cost cap (API Key Limit).
Filter using regular expressions to ensure that no sensitive information or personal information from the original document is included in the generated dataset.
Do not blindly trust model evaluation results alone, and always perform manual cross-checking on samples in parallel.

Lab deliverables

A question-answer evaluation set (JSONL file) with 100 or more items
Jupyter Notebook containing dataset generation and verification code
Quality analysis report including human review records

ASSIGNMENT

RAG Reliability Evaluation Set Regression Report

Deliverables

A dashboard analyzing the statistical distribution of the generated evaluation set (question length, answer length, document reference frequency, etc.)Results comparing two or more RAG configurations (e.g., search algorithm changes, model changes) using the same evaluation setError classification table (Hallucination, Contextual Irrelevance, etc.) and case analysis

Rubric

Does the evaluation set evenly reflect the content of the entire document corpus?
Is the synthetic data generation pipeline written in a reproducible form?
Is the error type classification specific and supported by quantitative evidence?
Has the validity of automated evaluation metrics been proven through human review?

KNOWLEDGE CHECK

Knowledge check

1What distinguishes the TrueTeacher methodology from existing synthetic data generation methods?It relies entirely on human-written summaries.It generates synthetic data by annotating various summaries generated by a model.It uses only small models as teacher models for training.It increases accuracy by writing datasets only manually.

2Why is factuality not determined solely by automated model evaluation when constructing a RAG evaluation set?Because automated model evaluation is too slow compared to humans.Because automated model evaluation is not perfect and cannot completely filter out hallucinations.Because human evaluation does not cost money.Because models are not needed for factuality evaluation.

FIELD CHECK

Completion criteria

Completed construction of a question-answer evaluation set with 100 or more items
Dataset quality analysis and submission of human review records
Implementation of notebook for RAG pipeline performance evaluation and creation of result report
Configuration of an evaluation package in a form that can be re-executed in a CI/CD environment

MODULE SOURCES

Module sources

[2305.11171] TrueTeacher: Learning Factual Consistency ... TrueTeacher: Learning Factual Consistency Evaluation with ... TrueTeacher: Generating Synthetic Data for Factual ... (PDF) TrueTeacher: Learning Factual Consistency Evaluation ... (PDF) TrueTeacher: Learning Factual Consistency Evaluation ... [PDF] TrueTeacher: Learning Factual Consistency Evaluation ...arxiv.org · paper