Add configurable NeMo Retriever RAGAS evaluation notebook - #2622
Add configurable NeMo Retriever RAGAS evaluation notebook#2622ChrisJar wants to merge 2 commits into
Conversation
Signed-off-by: Chris Jarrett <cjarrett@nvidia.com>
jperez999
left a comment
There was a problem hiding this comment.
Lets try and make this notebook, something you can set some variables for at the top and then do the run all cells command and just get the value at the end.
| "RETRIEVAL_K = 10\n", | ||
| "\n", | ||
| "# Set to True to create or replace the LanceDB index.\n", | ||
| "REBUILD_INDEX = False\n", |
There was a problem hiding this comment.
REBUILD_INDEX defaults to False, so following the default FinanceBench path skips ingestion and Retriever queries a table that does not exist. Default to building the index, or detect a missing table and ingest it.
| "\n", | ||
| "# Evaluation configuration\n", | ||
| "MAX_QUESTIONS = 50 # Set to None to evaluate every question.\n", | ||
| "NDCG_K = 10\n", |
There was a problem hiding this comment.
Please report document-level Recall and nDCG at 1, 3, 5, and 10. The current notebook calculates and displays only nDCG@10, which leaves the required retrieval cutoffs unavailable.
| "\n", | ||
| " ranked_documents = []\n", | ||
| " seen = set()\n", | ||
| " for hit in metadata[:k]:\n", |
There was a problem hiding this comment.
metadata[:k] applies the chunk cutoff before duplicate documents are collapsed. When early results contain repeated chunks, the metric scores fewer than k unique documents and may miss a relevant document that would enter document top-k after deduplication. Deduplicate candidate documents before the cutoff and retrieve enough candidates to fill the requested unique-document cutoffs.
| "EMBED_MODEL = \"nvidia/llama-nemotron-embed-vl-1b-v2\"\n", | ||
| "\n", | ||
| "# Evaluation configuration\n", | ||
| "MAX_QUESTIONS = 50 # Set to None to evaluate every question.\n", |
There was a problem hiding this comment.
Add explicit RUN_GENERATION and RUN_JUDGING switches, and make the final report work when either stage is disabled. It currently unconditionally depends on ragas_results. For bring-your-own data, document that document_field values must normalize to the same identifiers returned in retrieval metadata; otherwise document-level retrieval metrics cannot be calculated.
Description
Adds an end-to-end evaluation notebook under examples that:
Validation
pre-commit run --all-filesChecklist