Skip to content

Add configurable NeMo Retriever RAGAS evaluation notebook - #2622

Open
ChrisJar wants to merge 2 commits into
NVIDIA:mainfrom
ChrisJar:codex/add-nrl-ragas-notebook
Open

Add configurable NeMo Retriever RAGAS evaluation notebook#2622
ChrisJar wants to merge 2 commits into
NVIDIA:mainfrom
ChrisJar:codex/add-nrl-ragas-notebook

Conversation

@ChrisJar

@ChrisJar ChrisJar commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Description

Adds an end-to-end evaluation notebook under examples that:

  • provides guided FinanceBench setup or configurable custom dataset
  • ingests documents with the Retriever batch CLI
  • retrieves evidence and reports binary document nDCG@10 with duplicate source chunks collapsed
  • generates answers through the NVIDIA build endpoint with visible retries for transient API failures
  • evaluates answer accuracy, context relevance, and response groundedness with RAGAS
  • presents aggregate and per-question analysis without writing export artifacts

Validation

  • pre-commit run --all-files
  • notebook JSON and all Python code-cell syntax validated
  • confirmed the committed notebook contains no execution outputs or embedded API key
  • live end-to-end execution was not repeated because ingestion and evaluation require external models and API services

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes. Notebook structure and code syntax were validated; live service execution is documented above.
  • The documentation is up to date with these changes.

Signed-off-by: Chris Jarrett <cjarrett@nvidia.com>
@ChrisJar
ChrisJar requested review from a team as code owners September 1, 2026 19:29

@jperez999 jperez999 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets try and make this notebook, something you can set some variables for at the top and then do the run all cells command and just get the value at the end.

Comment thread examples/nrl_ragas.ipynb Outdated
"RETRIEVAL_K = 10\n",
"\n",
"# Set to True to create or replace the LanceDB index.\n",
"REBUILD_INDEX = False\n",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REBUILD_INDEX defaults to False, so following the default FinanceBench path skips ingestion and Retriever queries a table that does not exist. Default to building the index, or detect a missing table and ingest it.

Comment thread examples/nrl_ragas.ipynb Outdated
"\n",
"# Evaluation configuration\n",
"MAX_QUESTIONS = 50 # Set to None to evaluate every question.\n",
"NDCG_K = 10\n",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please report document-level Recall and nDCG at 1, 3, 5, and 10. The current notebook calculates and displays only nDCG@10, which leaves the required retrieval cutoffs unavailable.

Comment thread examples/nrl_ragas.ipynb Outdated
"\n",
" ranked_documents = []\n",
" seen = set()\n",
" for hit in metadata[:k]:\n",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

metadata[:k] applies the chunk cutoff before duplicate documents are collapsed. When early results contain repeated chunks, the metric scores fewer than k unique documents and may miss a relevant document that would enter document top-k after deduplication. Deduplicate candidate documents before the cutoff and retrieve enough candidates to fill the requested unique-document cutoffs.

Comment thread examples/nrl_ragas.ipynb Outdated
"EMBED_MODEL = \"nvidia/llama-nemotron-embed-vl-1b-v2\"\n",
"\n",
"# Evaluation configuration\n",
"MAX_QUESTIONS = 50 # Set to None to evaluate every question.\n",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add explicit RUN_GENERATION and RUN_JUDGING switches, and make the final report work when either stage is disabled. It currently unconditionally depends on ragas_results. For bring-your-own data, document that document_field values must normalize to the same identifiers returned in retrieval metadata; otherwise document-level retrieval metrics cannot be calculated.

@ChrisJar
ChrisJar requested a review from jperez999 September 10, 2026 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants