(diversity) Integrate determinant diversity in tokp full precision search and disk search by narendatha · Pull Request #1011 · microsoft/DiskANN

narendatha · 2026-05-04T12:30:38Z

Integrate determinant-diversity into topk full-precision search and disk search

Summary

Wires the determinant-diversity post-processing algorithm as an optional SearchPlugin for two search paths:

Async graph-index (full-precision topk via inmem::FullAccessor)
Disk-index (via DiskIndexSearcher post-processor)

Determinant-diversity reranks search results to increase geometric diversity among the top-k, at a configurable quality/speed tradeoff.

Motivation

Standard nearest neighbor search often returns highly similar/redundant results. For some scenarios like RAG search, it's more valuable to retrieve diverse, complementary documents that cover different aspects of the query. This PR implements a principled approach based on determinant maximization of the Gram matrix of query-scaled vectors.

Algorithm

The algorithm scales each candidate vector by (similarity_to_query)^power and then greedily selects vectors to maximize the determinant of the Gram matrix (equivalent to maximizing the volume of the parallelepiped spanned by the vectors).

Two variants are implemented:

eta = 0: Pure greedy orthogonalization (pivoted QR decomposition)
eta > 0: Ridge-regularized version that balances diversity with numerical stability

Both variants run in O(n k d) time where n = candidates, k = results to return, d = dimension.

Performance Characteristics

Tested on OpenAI embeddings dataset (3072 dimensions, 10K sample):

Metric	Normal Search	Det_Div Search	Notes
QPS	~150	~130	~13% overhead for diversity optimization
Recall@10	98-99%	87-88%	Expected trade-off for diversity

The recall reduction is expected - Det_Div search intentionally selects diverse vectors rather than the absolute nearest neighbors.

Diversity Improvement

We measure diversity using log-determinant of the Gram matrix of scaled vectors (higher = vectors span more volume = more diverse).

RAG search consistently improves diversity (all improvements >= 0)
Average improvement of +2.9 to +3.3 in log_det corresponds to 10-27x more volume spanned by selected vectors

Parameter Tuning Guide

`power` (default: 2.0)

Controls the trade-off between relevance and diversity:

Higher values (e.g., 3.0+): Strongly prefer relevance - vectors closer to the query get much higher weight, resulting in less diversity but higher recall
Lower values (e.g., 1.0): Prefer diversity over relevance - similarity weighting is more uniform, allowing more diverse vectors to be selected
Value of 0: Pure diversity maximization with no relevance weighting

`eta` (default: 0.01)

Ridge regularization parameter that controls numerical robustness and relevance preference:

Higher values (e.g., 0.1+): More robust numerically, but biases toward selecting high-relevance vectors over diverse ones
Lower values (e.g., 0.001): Closer to pure determinant maximization, maximizes diversity but may be less stable
Value of 0: Falls back to pure greedy orthogonalization (no ridge regularization)

Recommended Settings

Use Case	`power`	`eta`
Balanced (default)	2.0	0.01
High diversity	1.0	0.001
High relevance	3.0	0.1

Changes

Core algorithm (`diskann-providers`)

Added determinant_diversity_post_process() in diskann-providers/src/model/graph/provider/async_/determinant_diversity_post_process.rs
Two selection strategies controlled by eta:
- eta == 0: greedy orthogonalization
- eta > 0: eta-weighted residual selection

Async graph-index path (`diskann-benchmark`)

Added FullPrecisionVectorAccessor shim trait + impl for inmem::FullAccessor<f32, ...> so the post-processor can fetch real candidate vectors
Implemented DeterminantDiversity plugin that satisfies glue::SearchPostProcess using for<'a, 'b> SearchStrategy HRTB bound (credit: @hildebrandmw)

Disk-index path (`diskann-disk`, `diskann-benchmark`)

Added DeterminantDiversityAndFilter processor to DiskSearchPostProcessor enum
Extended searcher.search() to accept an optional SearchPostProcessorKind::DeterminantDiversity { power, eta }
Extended DiskSearchPhase JSON input to accept post_processor
Result-counting uses result_count from search stats to correctly bound post-processed results

Input/config

New TopkPostProcessor serde enum in diskann-benchmark/src/inputs/post_processor.rs with validation (power > 0, eta >= 0)
Two new example configs: async-determinant-diversity.json, disk-index-determinant-diversity.json

Benchmark JSON examples

// async graph-index
"post_processor": { "type": "determinant-diversity", "power": 1.0, "eta": 0.0 }

// disk-index
"post_processor": { "type": "determinant-diversity", "power": 1.0, "eta": 0.5 }

Notes

Determinant-diversity is only wired for full-precision topk on the async path; quantized (PQ/scalar) paths are intentionally excluded
Spherical, range, and filtered-search plugin variants are unchanged; this post-processor is topk-only

…ugins

Replace kind()-based string equality checks with explicit is_match() and get() phase-shape helpers on plugin structs. This avoids fragile ordering assumptions and makes each plugin responsible for recognising its own phase shape.

Co-authored-by: Copilot <copilot@github.com>

…plugins # Conflicts: # diskann-benchmark/src/backend/index/benchmarks.rs # diskann-benchmark/src/backend/index/product.rs # diskann-benchmark/src/backend/index/scalar.rs # diskann-benchmark/src/backend/index/search/plugins.rs # diskann-benchmark/src/backend/index/spherical.rs # diskann-benchmark/src/inputs/graph_index.rs # diskann-benchmark/src/inputs/mod.rs

Co-authored-by: Copilot <copilot@github.com>

…ojection issue This commit explores approaches to wire real candidate vectors into async determinant-diversity post-processing. Current state: IN COMPILATION ERROR (intentional for analysis) Attempted approaches: 1. Initial shim-trait FullPrecisionVectorAccessor with async get_full_precision_vector() - Resulted in 'implementation not general enough' at search_with() call 2. Removed explicit for<'a> post_processor::DeterminantDiversity bound - Still fails - the constraint is inherent in search_with() signature itself Root cause analysis: - search_with() requires: PP: for<'a> SearchPostProcess<S::SearchAccessor<'a>, T, O> - This means post-processor must work for ANY accessor lifetime 'a - But query = queries.row(query_idx) is borrowed for specific loop iteration lifetime - These are fundamentally incompatible - a borrowed value can't satisfy for<'a> generically Compiler errors (3 total): - 'not general enough': implementation needed for or<'a> but found specific '0 - 'does not live long enough': queries lifetime too short for 'static requirement Files modified: - diskann-benchmark/src/backend/index/benchmarks.rs: * Removed explicit for<'a> post_processor::DeterminantDiversity constraint * Narrowed plugin impl to FullPrecisionProvider<f32> - diskann-benchmark/src/backend/index/post_processor/determinant_diversity.rs: * Added shim trait FullPrecisionVectorAccessor * Async method get_full_precision_vector(&mut self, id) -> impl Future<...> Next steps to investigate: - Move determinant-diversity outside search_with() as post-processing reranking - This avoids HRTB entirely by applying after candidates are returned - Benchmark impact: measure recall/QPS with external reranking vs baseline Related context: - Disk index determinant-diversity works correctly (uses real vectors, shows 51-53% QPS cost) - Shared algorithm fixed (distance-to-similarity scoring direction) - Branch already merged with origin/main

Co-authored-by: Copilot <copilot@github.com>

…educe duplication - Use for<'a, 'b> SearchStrategy bound (user-provided fix) to break HRTB lifetime projection issue in the search_with post-processor constraint - Wire FullPrecisionVectorAccessor shim trait so async det-div post-processor fetches real candidate vectors instead of placeholder distances - Populate QPS/latency metrics in async det-div benchmark path (previously all 'missing') - Extract run_topk_timed helper to eliminate ~100 lines of duplicated loop/timing/recall machinery from DeterminantDiversity::run - Update async-determinant-diversity.json example tag (async-index-build -> graph-index-build) - Fix clippy::manual_async_fn in FullPrecisionVectorAccessor shim trait

codecov-commenter · 2026-05-06T05:22:20Z

Codecov Report

❌ Patch coverage is 73.62637% with 216 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.39%. Comparing base (d9ce362) to head (1807bf0).

Files with missing lines	Patch %	Lines
diskann-benchmark/src/inputs/graph_index.rs	8.88%	41 Missing ⚠️
.../src/index/post_processor/determinant_diversity.rs	0.00%	36 Missing ⚠️
diskann-benchmark/src/index/benchmarks.rs	16.66%	35 Missing ⚠️
diskann-benchmark/src/inputs/post_processor.rs	0.00%	27 Missing ⚠️
diskann-benchmark/src/index/search/plugins.rs	39.47%	23 Missing ⚠️
diskann-benchmark-core/src/search/graph/knn.rs	37.93%	18 Missing ⚠️
diskann-disk/src/search/provider/disk_provider.rs	94.02%	12 Missing ⚠️
diskann-benchmark/src/inputs/disk.rs	0.00%	8 Missing ⚠️
.../src/model/graph/provider/determinant_diversity.rs	97.65%	8 Missing ⚠️
diskann-providers/src/post_processor.rs	86.00%	7 Missing ⚠️
... and 1 more

❌ Your patch status has failed because the patch coverage (73.62%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1011      +/-   ##
==========================================
+ Coverage   89.47%   90.39%   +0.92%     
==========================================
  Files         486      490       +4     
  Lines       92161    92951     +790     
==========================================
+ Hits        82458    84025    +1567     
+ Misses       9703     8926     -777

Flag	Coverage Δ
miri	`90.39% <73.62%> (+0.92%)`	⬆️
unittests	`90.36% <73.62%> (+1.23%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-benchmark/src/index/inmem/spherical.rs	`100.00% <ø> (ø)`
diskann-benchmark/src/index/mod.rs	`100.00% <ø> (ø)`
diskann-benchmark/src/index/search/knn.rs	`77.77% <ø> (ø)`
diskann-benchmark/src/inputs/mod.rs	`81.25% <ø> (ø)`
diskann-disk/src/build/builder/core.rs	`95.35% <100.00%> (+<0.01%)`	⬆️
diskann-tools/src/utils/search_disk_index.rs	`0.00% <0.00%> (ø)`
diskann-providers/src/post_processor.rs	`86.00% <86.00%> (ø)`
diskann-benchmark/src/inputs/disk.rs	`1.50% <0.00%> (-0.07%)`	⬇️
.../src/model/graph/provider/determinant_diversity.rs	`97.65% <97.65%> (ø)`
diskann-disk/src/search/provider/disk_provider.rs	`93.71% <94.02%> (+0.03%)`	⬆️
... and 6 more

... and 39 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Use for<'a, 'b> SearchStrategy bound to resolve HRTB lifetime mismatch - Wire FullPrecisionVectorAccessor shim trait for async det-div to fetch real vectors - Implement DeterminantDiversity post-processor for async graph-index path - Extract run_topk_timed helper to eliminate ~100 lines of code duplication - Wire post_processor parameter to disk-index search pipeline - Update search parameter handling and result counting for post-processed results - Add TopkPostProcessor input type and necessary imports - Populate QPS/latency metrics in async det-div benchmark path

…rosoft/DiskANN into u/narendatha/det_div_plugins

Co-authored-by: Copilot <copilot@github.com>

- Remove duplicate post_process_with_eta_f32 and post_process_greedy_orthogonalization_f32 - Unify into single greedy_orthogonal_select() with inv_sqrt_eta parameter - inv_sqrt_eta=1.0/sqrt(eta) for ridge-regularized path (eta>0) - inv_sqrt_eta=1.0 for exact greedy path (eta=0) - Eliminates ~80 lines of near-duplicate code - All 14 existing tests continue to pass

- Use contiguous Matrix allocation instead of separate Vec per candidate - Reduces heap allocations from O(n) to O(1) where n = num candidates - Improves cache locality during orthogonalization iteration - Access residuals via row(i) / row_mut(i) instead of indexing Vec of Vecs - All 14 tests continue to pass

- Move implementation to model/graph/provider/determinant_diversity.rs - Export via provider::mod.rs as determinant_diversity_post_process - Keep backward-compatible re-export from async_::mod.rs - Remove old async_-scoped source file - Verify build and full diskann-providers tests pass

…plugins

arkrishn94

Thanks Naren, I am glad this feature is starting to land in the repo. Left a few small comments here but my main question is whether greedy_orthogonal_select should be written in a way that is more pluggable and generalizable -

Allocation and format of full precision vectors. Isn't Vec<(Id, Vec<f32>)> too restrictive?
Should we lean into the matrix API more and its optimized implementations in diskann-quantization?

…plugins # Conflicts: # diskann-benchmark/src/backend/index/search/knn.rs # diskann-benchmark/src/inputs/graph_index.rs # diskann-benchmark/src/inputs/mod.rs

…e Positive<f32> for power

…rrors Indented blocks in /// comments are interpreted as Rust doctests by rustdoc; the formula lines (e.g. 'alpha_i = ...', 'similarity(d) = ...') were being compiled and failed in CI. Wrap them in ext fences so they remain prose.

…plugins # Conflicts: # diskann-disk/src/search/provider/disk_provider.rs

…plugins # Conflicts: # diskann-benchmark/src/inputs/graph_index.rs

hildebrandmw

Thanks Naren, this has been a journey. The PR still has a lot of little structural issues that need to be addressed before merging. I've done my best to enumerate the issues and potential fixes in the inline comments. Please read through all the comments first as some of them are inter-related.

Note that this review focused on architecture and integration. I'd like to take one more look at the algorithm when the structural issues are resolved.

Feel free to reach out with any questions!

…rix-based interface - Updated diskann-providers determinant_diversity to accept MutMatrixView instead of Vec<(Id, f32, Vec<f32>)> - Maintains identical algorithmic behavior and recall across all test datasets (SIFT Small, Enron) - Improves memory layout efficiency with contiguous distance/ID arrays - Updated all call sites: diskann-benchmark inmem and diskann-disk providers - Verified with comprehensive A/B benchmarks: * Recall parity: 100% match vs archived baselines * SIFT Small (256 vectors): -45% QPS (small dataset regression) * Enron (46,750 vectors): -5% to +13% QPS (larger dataset improvement) - All tests passing, no panics or runtime errors

Replaces the previous post-processor sub-enum pattern with a dedicated TopkDeterminantDiversity variant on the SearchPhase enum, following the established pattern for Range, BetaFilter, and MultiHopFilter variants. - Add TopkDeterminantDiversityPhase struct with validate() that reuses DeterminantDiversityParams::new for power/eta validation - Add TopkDeterminantDiversity variants to SearchPhase and SearchPhaseKind - Wire plugins::DeterminantDiversity to use the new variant - Remove now-redundant post_processor field from TopkSearchPhase - Update example config (async-determinant-diversity.json) to new format - DiskSearchPhase intentionally unchanged (separate code path) Addresses PR #1011 review comment r3390077824.

…r dispatch Replace the two Search impls on KNN (one for () default, one for explicit PP) and the Option<PP> + expect(...) panic with a single AsPostProcessor trait. Defaulted resolves to the strategy's default post-processor; Forwarded<PP> carries an explicit one. KNN::new returns KNN<.., Defaulted>; KNN::with_postprocessor returns KNN<.., Forwarded<PP>>. The ConfiguredPostProcessor marker trait is removed.

…bility Remove the brittle hardcoded checks in JSON-input validation that bailed when topk-determinant-diversity was paired with non-Float32 / PQ / SQ / spherical / streaming graph builds. The plugin registry already provides the same compatibility guarantee at run time (only Float32 FullPrecision registers DeterminantDiversity), and produces a generic 'Unsupported search phase' error listing the kinds the backend actually supports.

Adds test_disk_search_determinant_diversity exercising the new SearchPostProcessorKind::DeterminantDiversity path through DiskIndexSearcher::search end-to-end. The test asserts: result count matches k, all selected ids are a subset of the L=20 candidate pool, no duplicates, top-1 matches the nearest neighbor (greedy invariant), pure-greedy (eta=0) preserves the candidate-pool subset invariant, and the vector_filter is honored by DeterminantDiversityAndFilter.

…plugins # Conflicts: # diskann-benchmark/src/backend/index/benchmarks.rs # diskann-benchmark/src/inputs/graph_index.rs

…plugins # Conflicts: # diskann-benchmark/src/inputs/graph_index.rs

…plugins # Conflicts: # diskann-benchmark/src/disk_index/search.rs # diskann-benchmark/src/index/benchmarks.rs # diskann-benchmark/src/index/search/plugins.rs

Mark Hildebrand and others added 12 commits April 28, 2026 14:47

Make benchmarks stateful.

dcdc377

Checkpoint.

d60ac07

Checkopint.

6cc92bd

Oh boy.

f050e3d

Merge remote-tracking branch 'origin/main' into mhildebr/benchmark-pl…

2d03d11

…ugins

Fix delimiters.

aa78836

Revert formatting.

ccf891a

Make plugins slightly more flexible.

2e060ea

Merge branch 'main' into mhildebr/benchmark-plugins

7a85df8

Add determinant-diversity plugin support on search-plugin architecture

fc8008a

Integrate determinant-diversity via disk search_with post-processor

298d1b8

Restrict determinant-diversity to async full-precision topk

fbd34fd

narendatha changed the base branch from main to mhildebr/benchmark-plugins May 4, 2026 12:41

Keep single async determinant-diversity example JSON

851174e

Base automatically changed from mhildebr/benchmark-plugins to main May 5, 2026 00:45

narendatha and others added 10 commits May 5, 2026 12:41

Use SearchPhaseKind::as_str in benchmark plugin kinds

3a6aa1a

remove serde defaults

ccfe4d7

Co-authored-by: Copilot <copilot@github.com>

minor merge fix

86883b7

hook up actual algorithm, replace placeholder.

554bc7f

Co-authored-by: Copilot <copilot@github.com>

apply mark's beautiful fix for lifetime issue

b73abc8

Co-authored-by: Copilot <copilot@github.com>

Fix CI clippy-features spherical plugin errors and apply formatting

701ce8e

narendatha and others added 4 commits May 6, 2026 16:57

Merge branch 'u/narendatha/det_div_plugins' of https://github.com/mic…

6f47ba3

…rosoft/DiskANN into u/narendatha/det_div_plugins

imrpove code coverage

8acbba2

minor fix

a48e255

Co-authored-by: Copilot <copilot@github.com>

narendatha added 10 commits May 13, 2026 13:20

Refactor determinant-diversity benchmark path

ca20a24

cargo fmt and clippy fixes for CI

f538583

Use shared determinant-diversity params validation

f58789d

Merge remote-tracking branch 'origin/main' into u/narendatha/det_div_…

79c635f

…plugins

code review comment, use a struct instead of a tuple

3c11d36

Refine determinant-diversity invariants and range representation

b65e673

minor code cleanup

85797ce

arkrishn94 reviewed May 26, 2026

View reviewed changes

narendatha added 7 commits June 2, 2026 10:06

Merge remote-tracking branch 'origin/main' into u/narendatha/det_div_…

8aeee5a

…plugins # Conflicts: # diskann-benchmark/src/backend/index/search/knn.rs # diskann-benchmark/src/inputs/graph_index.rs # diskann-benchmark/src/inputs/mod.rs

Address PR #1011 r3268094250: document determinant_diversity math; us…

19380a7

…e Positive<f32> for power

knn: route configured post-processors through KNN::with_postprocessor

63f1ab8

det-div: add unit tests for edge cases and selection invariants

2f482f9

Merge remote-tracking branch 'origin/main' into u/narendatha/det_div_…

bc03532

…plugins # Conflicts: # diskann-disk/src/search/provider/disk_provider.rs

Merge remote-tracking branch 'origin/main' into u/narendatha/det_div_…

ce8db8d

…plugins # Conflicts: # diskann-benchmark/src/inputs/graph_index.rs

hildebrandmw reviewed Jun 10, 2026

View reviewed changes

narendatha added 11 commits June 12, 2026 12:41

minor fixes.

125ae1b

Mark's code review comments: minor refactors

98b6a7d

Merge remote-tracking branch 'origin/main' into u/narendatha/det_div_…

b416b95

…plugins # Conflicts: # diskann-benchmark/src/backend/index/benchmarks.rs # diskann-benchmark/src/inputs/graph_index.rs

fix test failure

11c989e

Merge remote-tracking branch 'origin/main' into u/narendatha/det_div_…

855fbd2

…plugins # Conflicts: # diskann-benchmark/src/inputs/graph_index.rs

Merge remote-tracking branch 'origin/main' into u/narendatha/det_div_…

1807bf0

…plugins # Conflicts: # diskann-benchmark/src/disk_index/search.rs # diskann-benchmark/src/index/benchmarks.rs # diskann-benchmark/src/index/search/plugins.rs

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(diversity) Integrate determinant diversity in tokp full precision search and disk search#1011

(diversity) Integrate determinant diversity in tokp full precision search and disk search#1011
narendatha wants to merge 61 commits into
mainfrom
u/narendatha/det_div_plugins

narendatha commented May 4, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented May 6, 2026 •

edited

Loading

Uh oh!

arkrishn94 left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hildebrandmw left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

narendatha commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integrate determinant-diversity into topk full-precision search and disk search

Summary

Motivation

Algorithm

Performance Characteristics

Diversity Improvement

Parameter Tuning Guide

power (default: 2.0)

eta (default: 0.01)

Recommended Settings

Changes

Core algorithm (diskann-providers)

Async graph-index path (diskann-benchmark)

Disk-index path (diskann-disk, diskann-benchmark)

Input/config

Benchmark JSON examples

Notes

Uh oh!

codecov-commenter commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

arkrishn94 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hildebrandmw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

narendatha commented May 4, 2026 •

edited

Loading

`power` (default: 2.0)

`eta` (default: 0.01)

Core algorithm (`diskann-providers`)

Async graph-index path (`diskann-benchmark`)

Disk-index path (`diskann-disk`, `diskann-benchmark`)

codecov-commenter commented May 6, 2026 •

edited

Loading

arkrishn94 left a comment •

edited

Loading