Advanced RAG and Evaluation Specialist

Analysis of RAG failure modes: missing, wrong, stale or conflicting context. This module defines these failure types, shows how to detect and diagnose them, and equips practitioners to pinpoint sources of error in retrieval-augmented generation pipelines.

Defines contextual failure categories — missing, wrong, stale, conflicting — and how each affects generated outputs.

Details observable signals and diagnostic steps to separate retrieval failures from generation issues and prioritize fixes.

Provides a structured method to record and reproduce failures so teams can iterate on RAG components effectively.

Advanced retrieval patterns for RAG: query rewriting and decomposition, HyDE, multi-query/multi-index retrieval, reranking and contextual compression. The unit explains these techniques and how they improve relevance and document coverage.

Covers query rewriting, decomposition and HyDE as practical ways to produce stronger retrieval queries.

Explains multi-query and multi-index setups to span diverse subdomains or heterogeneous corpora.

Describes reranking and contextual compression techniques to reduce noise and surface the most useful evidence for generation.

Evaluation pipelines for RAG: synthetic dataset generation, retrieval metrics (recall, precision, MRR, nDCG) and generation metrics (faithfulness, helpfulness, correctness), plus LLM-as-judge calibration and bias. Practical approach to measure and compare systems.

Shows how to create synthetic datasets tailored to RAG test cases for reproducible diagnostics.

Defines retrieval metrics — recall, precision, MRR, nDCG — and how to interpret them for coverage and ranking evaluation.

Covers generation metrics focused on faithfulness and usefulness, and discusses using an LLM as judge including calibration challenges and bias considerations.

Continuous improvement for RAG systems: regression tests, human feedback and annotation workflows, experiment tracking and release gates, and reporting to product and business stakeholders.

Details regression-test design to monitor RAG quality and prevent unintended regressions from changes.

Covers human feedback loops and annotation workflows to gather quality signals and prioritize fixes.

Explains experiment tracking, release gate definition, and how to structure clear reports for product and business stakeholders.