CURRICULUM 1.0.0+40bb735b5f50 · intermediate

AI & Software

RAG Evaluation and Reliability Experiments

Learners will be able to evaluate the retrieval and generation stages of a RAG pipeline separately and design a quantitative regression testing framework that utilizes automated evaluation tools like Ragas to reduce hallucinations and verify factuality.

  • retrieval-augmented-generation
  • rag-evaluation
  • information-retrieval
  • llm-reliability
TIME96.0h
MODULES10
BUDGET₩0–100000
REVIEWED2026-08-30

YOUR FIELD LOG

Learning progress

0% complete

Create a fixed document set and question/answer evaluation set, then quantitatively measure retrieval quality, grounded faithfulness, citation accuracy, and failure modes to complete a reproducible RAG regression evaluation report.

Learners will be able to evaluate the retrieval and generation stages of a RAG pipeline separately and design a quantitative regression testing framework that utilizes automated evaluation tools like Ragas to reduce hallucinations and verify factuality.

STUDY MAP

Curriculum

1

RAG Foundations and Dataset Construction

  1. Understanding RAG Architecture8.0 hours · Understand the core components of the RAG (Retrieval-Augmented Generation) architecture.
  2. Evaluation Document Corpus Curation8.0 hours · Understand the importance of a fixed document corpus for RAG evaluation.
  3. Question-Answer Evaluation Set Construction8.0 hours · Understand the necessity of a high-quality Question-Answer (QA) evaluation set for RAG system evaluation.
2

Retrieval and Generation Quality Evaluation

  1. Search Quality Metrics (Recall @k, MRR, nDCG)8.0 hours · Understand the importance of the retrieval step in a Retrieval-Augmented Generation (RAG) pipeline.
  2. Faithfulness Evaluation8.0 hours · Understand the concept of Faithfulness and identify its importance in RAG systems.
  3. Citation Accuracy and Source Tracking8.0 hours · Understand how faithfully a response reflects the content of retrieved documents in a RAG system.
  4. Applying Automated Evaluation Framework (Ragas)10.0 hours · Understand the core evaluation dimensions (retrieval and generation quality) of a RAG pipeline.
3

Reliability Verification and Reporting

  1. Failure Type Classification and Error Analysis10.0 hours · Able to identify and classify failure types occurring in RAG systems.
  2. Sample Human Review and Comparison10.0 hours · Understand the gap between automated RAG evaluation metrics and actual factual accuracy.
  3. Reproducible Regression Evaluation Reporting18.0 hours · Understand the framework for reproducible regression evaluation of RAG systems.

FINAL BUILD

RAG System Reliability Regression Evaluation Report

Deliverables

Rubric

  • Traceability of the evaluation set sources and precision of question-document correspondence.
  • Validity of quantitative measurement of retrieval and generation quality metrics.
  • Qualitative agreement analysis between automatic evaluation results and human review results.
  • Clear classification and analysis of failure types (retrieval failure vs. generation failure).
  • Reproducibility of the report and CI integration capability of the regression evaluation framework.

Safety criteria

  • Compliance with measures to prevent the leakage of personal information during the evaluation process.
  • Safe data handling against prompt injection.
  • Parallel implementation of a human review process for fact-checking.
  • Safe API usage through cost and request limits.

RESEARCH LEDGER

Sources

  1. Stanford CS 224N | Natural Language Processing with Deep Learningweb.stanford.edu · university · 2026-08-30
  2. Natural Language Processing with Deep Learning CS224N/Ling284web.stanford.edu · university · 2026-08-30
  3. [2309.15217] Ragas: Automated Evaluation of Retrieval ... Ragas: Automated Evaluation of Retrieval Augmented Generation RAGAS: Automated Evaluation of Retrieval Augmented Generation RAGAs: Automated Evaluation of Retrieval Augmented Generation RAGAS: Automated Evaluation of Retrieval Augmented Generation RAGAS: Automated Evaluation of Retrieval Augmented Generationarxiv.org · paper · 2026-08-30
  4. Ragas: Automated Evaluation of Retrieval Augmented Generationarxiv.org · paper · 2026-08-30
  5. [2305.11171] TrueTeacher: Learning Factual Consistency ... TrueTeacher: Learning Factual Consistency Evaluation with ... TrueTeacher: Generating Synthetic Data for Factual ... (PDF) TrueTeacher: Learning Factual Consistency Evaluation ... (PDF) TrueTeacher: Learning Factual Consistency Evaluation ... [PDF] TrueTeacher: Learning Factual Consistency Evaluation ...arxiv.org · paper · 2026-08-30