CAGRA-ACE: Make Disk Workspaces Transactional and Non-destructive

[julianmi]

Make ACE disk workspaces non-destructive and failure-safe.

• Create CAGRA artifacts exclusively and remove a newly created artifact if NumPy preallocation or header writing fails.
• Document the C and C++ build_dir contract: named artifacts must be absent and concurrent builds need separate directories.
• Stage HNSW output and publish hnsw_index.bin only when that name is absent. A completed CAGRA stage remains if later HNSW conversion fails.
• Add C++ regressions for partial-file cleanup and preserving an existing HNSW index.

[dantegd]

Good PR!

Had one correctness comment only, and a couple of questions besides the review one:

• Could we mirror the new build_dir contract in c/include/cuvs/neighbors/cagra.h? That public API exposes the same ACE parameters, but its documentation still doesn’t mention that named artifacts must be absent or that simultaneous builds require distinct directories.
• Wasn't sure what is intended transaction boundary here? A public hnsw::build using ACE first creates and commits the four CAGRA artifacts, then writes hnsw_index.bin into the same build_dir. The latter is still opened with truncation in hnsw.hpp. Is this PR’s non-destructive contract intended to cover only the CAGRA stage, or the complete HNSW ACE build seen by the caller? If it is CAGRA only, could we state that scope explicitly?

[dantegd]

Should we make the exclusive create branch unlink path if anything after this successful open throws? With exclusive=true, open creates the file here, but posix_fallocate or write can still fail before create_numpy_file returns. The caller therefore never reaches mark_artifact_created, so rollback leaves a partial ACE artifact that blocks the next retry.

What do you think about guarding the newly created path inside this helper until the function succeeds, while preserving the current non-exclusive behavior?

[julianmi]

Good catch, thanks. After an exclusive open succeeds, create_numpy_file now closes and unlinks the newly created path if preallocation, seek, or header writing throws. The non-exclusive truncating behavior is unchanged. I've also added a regression test.

[julianmi]

Could we mirror the new build_dir contract in c/include/cuvs/neighbors/cagra.h? That public API exposes the same ACE parameters, but its documentation still doesn’t mention that named artifacts must be absent or that simultaneous builds require distinct directories.

Agreed. cuvsAceParams maps to the same disk workspace, so its public contract should match the C++ API. I've
updated the documentation.

Wasn't sure what is intended transaction boundary here? A public hnsw::build using ACE first creates and commits the four CAGRA artifacts, then writes hnsw_index.bin into the same build_dir. The latter is still opened with truncation in hnsw.hpp. Is this PR’s non-destructive contract intended to cover only the CAGRA stage, or the complete HNSW ACE build seen by the caller? If it is CAGRA only, could we state that scope explicitly?

Yes, I think we should extend it to HNSW ACE builds as well. I've added an exclusive_hnsw_output_file helper class. hnsw_index.bin is linked to the target path when the CAGRA to HNSW conversion completes. Otherwise, the staging file is removed. I propose to leave the CAGRA builds in case of an HNSW index conversion failure such that a user can repeat hnsw::from_cagra only. I've also added a regression test. What do you think?