feat(gluten): export a Lance fragment scan as an Arrow C stream

[sezruby]

What

Adds LanceArrowStreamScanner to the base module: it plans a Lance fragment scan through the existing LanceFragmentScanner and re-exports the scan's ArrowReader as an Arrow C Data Interface stream (ArrowArrayStream).

Only the stream's C-struct address (LanceArrowStream#streamAddress()) crosses the JVM/native boundary, so a native consumer such as Apache Gluten / Velox does not need to match lance-spark's Arrow build or classloader.

Why

This is step 2 of the JVM Arrow-in/Arrow-out read path for Gluten support (apache/gluten#12263, Approach 2): lance-spark hands a native engine an Arrow C stream over a planned fragment scan. A gluten-side scan-offload rule that consumes streamAddress() is the remaining piece.

Details

• All scan planning — column projection, filter pushdown, limit/offset, row-id/row-address, batch size — is delegated to LanceFragmentScanner, so the exported stream yields the same rows in the same order as the Spark columnar reader.
• The scan is materialized one Arrow batch at a time as the consumer pulls, so peak JVM Arrow memory is bounded by a single batch rather than the whole fragment.
• The returned LanceArrowStream handle owns the exported stream and the scanner behind it. exportArrayStream transfers ownership of the reader to the stream, so close() releases the stream (which closes the reader via the C release callback) and then the scanner and dataset — it never closes the reader itself.
• A TODO(lance#7259) marks a one-line swap to LanceScanner#exportArrowStream(long) once that lands upstream: the Rust core would populate the caller's stream directly and skip the JVM-side Arrow materialization. The LanceArrowStream contract is unchanged, so no consumer needs to be touched.

Test

LanceArrowStreamScannerTest exports each fragment of the bundled test table, re-imports it via Data.importArrayStream (standing in for a native consumer), drains it, and asserts the rows match the Spark columnar reader. Closing the imported reader and then the handle under the leak-checking RootAllocator also verifies the export/reader/scanner lifecycle releases cleanly.

mvn -pl lance-spark-base_2.12 -am -DskipTests install
mvn -pl lance-spark-base_2.12 test -Dtest=LanceArrowStreamScannerTest

🤖 Generated with Claude Code