Search Quality Metrics (Recall @k, MRR, nDCG)

The Importance of RAG Search Quality Evaluation

RAG systems retrieve relevant information from external databases and pass it to an LLM to generate answers [S3]. Therefore, if the search stage fails to identify highly relevant and focused context, even a powerful LLM will struggle to generate accurate answers [S3]. Evaluating search quality is the first step toward improving the overall performance of the RAG architecture.

Key Search Evaluation Metrics

1. Recall @k: Measures whether the actual correct answer is included in the top k retrieved results. In other words, it is a metric to confirm whether the necessary information was captured by the search system.
2. MRR (Mean Reciprocal Rank): Measures the position of the correct answer (relevant document) for a user query within the search result list. The more the relevant document appears in the first position, the closer the MRR value is to 1, resulting in a higher score.
3. nDCG (normalized Discounted Cumulative Gain): A metric that considers the ranking of search results; it assigns higher scores as more relevant documents are positioned toward the top. It evaluates the ‘ranking accuracy’ of search results more precisely than simple inclusion (Recall).

These metrics are essential for system improvement when ground truth is available, and frameworks like Ragas provide tools to quantitatively analyze these dimensions [S3, S4].