Add PyLucene backend for cuVS-Lucene vector search in cuvs-bench

[nvzm123]

Summary

This PR adds a built-in pylucene backend to cuVS Bench for building and searching local Lucene vector indexes through PyLucene and cuVS-Lucene.

It supports:

• Lucene101AcceleratedHNSWCodec for cuVS-assisted HNSW construction with Lucene HNSW search;
• CuVS2510GPUSearchCodec for GPU CAGRA construction and search;
• packaged configurations, deterministic parameter sweeps, automatic tuning, dry runs, index reuse, forced rebuilds, query batching, and standard cuVS Bench CSV results; and
• h5py as an explicit dependency for the existing dataset-preparation path.

HNSW parameters and topology

The HNSW backend exposes these benchmark parameters:

• m maps to AcceleratedHNSWParams.maxConn;
• ef_construction maps to AcceleratedHNSWParams.beamWidth;
• both inputs use cuVS-Lucene's SAME_GRAPH_FOOTPRINT heuristic to derive the underlying CAGRA build parameters;
• num_candidates controls Lucene's KnnFloatVectorQuery candidate budget and may be swept independently for each built index; and
• direct_single_segment requests one direct Lucene segment without a force merge and verifies the committed topology.

Stock PyLucene instantiates codecs through no-argument constructors. The backend therefore compiles a small Java adapter before JVM startup that delegates to the production accelerated codec with the requested heuristic parameters. It does not reimplement the codec.

Automatic tuning covers m, ef_construction, and num_candidates, with the candidate lower bound resolved from top_k. Build parameters and segment count are recorded in schema-v4 provenance so incompatible indexes cannot be silently reused.

Runtime and integrity behavior

The backend validates the PyLucene 10.2 ABI, JAR layout, native-library paths, Lucene SPI codecs, dataset shape, and index location before execution. PyLucene and the JVM are initialized lazily.

HNSW preserves cuVS-Lucene's production behavior: it uses the accelerated writer when available and intentionally falls back to Lucene's CPU writer otherwise. CAGRA requires GPU support and is validated fail-closed. CAGRA segment metadata, vector dimensions and counts, file coverage, headers, footers, and checksums are verified before results are accepted.

Atomic, commit-bound provenance protects index reuse. Failed new builds remove only their partial output, and preflight failures preserve an existing index. In-process backend results use the shared direct CSV exporter and preserve build/search identity, latency percentiles, and failure handling.

Requirements and limits

• FLOAT32 Euclidean/L2 datasets, at most 4096 dimensions, and at least two indexed vectors
• latency mode with one search thread; query batching through --batch-size
• num_candidates >= top_k for HNSW; this is Lucene's candidate budget, not a direct cuVS ef_search setting
• CAGRA searches with k <= 1024
• direct_single_segment remains subject to Lucene's per-indexing-thread hard RAM limit and fails if Lucene commits more than one segment
• process-wide PyLucene JVM settings cannot change after initialization

PyLucene 10.2 remains a source-built external dependency. This PR depends on the in-tree PyLucene support in NVIDIA/cuvs#2475 until that PR is merged. Exact tested revisions and artifact provenance are recorded under Validation. PR #2475 ports the PyLucene work originally developed in NVIDIA/cuvs-lucene#174; no standalone cuvs-lucene checkout or patch composition is required.

Test coverage

Pytest coverage includes backend registration, configuration expansion and tuning, parameter validation, adapter compilation and classpath failures, JVM and SPI validation, index build/reuse/cleanup, single- and multi-segment topology, search candidate propagation, result export, provenance mismatches, CAGRA corruption detection, and CLI failure handling.

The opt-in live suite exercises both production codecs with a real PyLucene/JVM/cuVS runtime. It proves configured accelerated-writer selection, intentional CPU fallback in a fresh process, sequential parameter isolation, direct single-segment construction, num_candidates wiring, CAGRA validation, index reuse, search correctness, and CLI build/search execution.

Validation

Validation was performed against these revisions, which were current when the run completed:

• cuVS Bench PR Add PyLucene backend for cuVS-Lucene vector search in cuvs-bench #2385: f993daa9d14106477ebfcb8b3f90541d589669e3
• cuVS dependency PR Add PyLucene integration and CPU/GPU end-to-end tests #2475: ecdff3234053b43706a0021a5464b6ab1a38be76

The native cuVS libraries, cuvs-java JAR, and cuvs-lucene JAR used below were built from the same #2475 checkout. Before the downstream run, an import and native-map check confirmed that Python loaded cuvs_bench.backends.pylucene from the #2385 source tree while the four cuVS/RMM native libraries came from the #2475 build tree.

Upstream dependency validation

• From Add PyLucene integration and CPU/GPU end-to-end tests #2475 java/cuvs-lucene, python -m pytest -q -s src/test/python/test_pylucene_end_to_end.py: 24 passed in 177.34s

This suite validates #2475's PyLucene integration independently. It does not exercise the cuVS Bench backend.

Downstream cuVS Bench integration validation

• From Add PyLucene backend for cuVS-Lucene vector search in cuvs-bench #2385 python/cuvs_bench, CUVS_BENCH_PYLUCENE_INTEGRATION=1 python -m pytest -q -s -m pylucene cuvs_bench/tests/pylucene/test_pylucene_integration.py, with the exact Add PyLucene integration and CPU/GPU end-to-end tests #2475 JAR and native-library paths exported: 23 passed in 37.89s

This suite imports and executes #2385's PyLuceneBackend and cuVS Bench orchestration against artifacts built from #2475. It includes a fresh-process python -m cuvs_bench.run build/search sweep and validates cuVS Bench CSV output, so it tests the downstream integration rather than only the dependency.

Regression and static validation

• Full python/cuvs_bench pytest suite in the dedicated GPU regression environment: 487 passed, 49 skipped in 72.56s
• Ruff lint and format checks for all 23 Python files changed by this PR, using the monorepo root configuration: passed
• Worktree and full PR git diff --check: passed

Related work

• NVIDIA/cuvs#2475
• NVIDIA/cuvs#1907
• NVIDIA/cuvs-lucene#174 (historical source of the in-tree PyLucene port)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.