RAG evaluation: separate retrieval from answer quality

01 / Two stages in retrieval-augmented generation

Parse documents, create chunks with source metadata and build an index. At query time retrieve candidate passages, optionally rerank them, and supply the question and evidence to a generator. RAG changes inference context and usually does not update generator weights.

Dense retrieval matches semantic paraphrases; BM25 may handle exact identifiers and product codes better. Hybrid retrieval can broaden recall, but any benefit must be measured on the same question set.

02 / Build a small evaluation set first

Record a query ID, question, acceptable source IDs, answer requirements and answerability. Include exact lookup, multi-passage reasoning, version conflicts and missing information. Freeze the corpus snapshot, chunking, k and development/test split before comparison; keep paraphrases of one question in the same split.

Labels may be document IDs or passage IDs, but predictions and labels must use the same unit. This example deduplicates document IDs before taking k. If the real constraint is a passage budget, define and report that budget separately.

03 / Calculate and run retrieval metrics

Recall@k divides relevant sources found in the first k results by all labeled relevant sources. Reciprocal rank is the inverse rank of the first relevant result; averaging it over queries gives MRR. Questions with no relevant sources need a separate abstention evaluation.

From the repository root run python notebook/examples/rag_metrics.py. The example prints 0.5 for Recall@2 and reciprocal rank. These values illustrate the formulas, not model performance.

"""Toy retrieval metrics. No model or network access is needed."""
def recall_at_k(ranked, relevant, k):
    if k < 1:
        raise ValueError("k must be positive")
    relevant = set(relevant)
    if not relevant:
        return None  # Unanswerable queries need a separate abstention measure.
    # Remove duplicate document IDs before applying the retrieval budget.
    retrieved = set(list(dict.fromkeys(ranked))[:k])
    return len(retrieved & relevant) / len(relevant)

def reciprocal_rank(ranked, relevant):
    relevant = set(relevant)
    for rank, doc in enumerate(dict.fromkeys(ranked), start=1):
        if doc in relevant:
            return 1 / rank
    return 0.0

if __name__ == "__main__":
    ranked = ["policy-old", "guide", "guide", "policy-current"]
    gold = {"guide", "policy-current"}
    print("Recall@2:", recall_at_k(ranked, gold, 2))  # 0.5
    print("Reciprocal rank:", reciprocal_rank(ranked, gold))  # 0.5

04 / A citation must actually support the answer

Break an answer into checkable claims: does the citation resolve, does the passage support the claim, are conditions or dates missing, and does the system acknowledge missing information? Start with human review on a small sample; an automated judge needs calibration against those judgments.

Retrieved pages and documents are untrusted data: their instructions must not replace application rules. For internal documents, enforce access scope at retrieval and content delivery; cache keys must account for that scope too.

05 / Engineering reflection and next exercise

When an answer fails, preserve retrieval results and citations, then classify the cause as missing evidence, retrieval failure, ranking failure or unsupported generation. Compare BM25 with one dense retriever before changing chunking or reranking; log latency, cost and failure cases. These are proposed exercises, not completed experiments.