Evaluation Document Corpus Curation

Corpus Curation for RAG Evaluation

Large Language Models (LLMs) carry a risk of Hallucination when relying on knowledge within learned parameters. Retrieval-Augmented Generation (RAG) overcomes this limitation by allowing models to access external knowledge in real-time [S2]. A Fixed Evaluation Corpus that is controlled and stable is essential to quantitatively evaluate the reliability of an effective RAG system.

1. Corpus Quality Determination Factors

• Factuality: Information within documents must be up-to-date and factual. A corpus containing incorrect information leads to the generation of incorrect answers.
• Domain Suitability: It must include topics and vocabulary similar to the actual service environment being evaluated.
• De-duplication: If identical information is duplicated across multiple documents, it hinders retrieval result diversity and introduces bias into evaluation statistics.

2. Building QA Evaluation Datasets

Evaluation is impossible with only a document corpus. You must build ‘Document-Question-Answer’ pairs to measure whether the retriever brings in relevant documents and whether the generator provides accurate answers based on them.

• Question Generation: Automatically generate questions from documents using an LLM, or have them written directly by domain experts.
• Answer Definition: Clearly specify the document segment (Chunk) that serves as the evidence for the answer.

3. Data Splitting and Integrity

For evaluation set reliability, you must strictly split the data into a Development Set and a Hold-out Test Set. You must prevent ‘data leakage’, where questions in the evaluation set are directly exposed because they are contained in the retrieval target documents.