[lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter

[qzyu999]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Paimon's lake writer implements SupportsRecordBatchWrite (landed in #3417/#3418), enabling direct Arrow batch to Parquet writes for append-only tables. This avoids the row-by-row serialization overhead through GenericRecord objects.

The Iceberg lake writer (IcebergLakeWriter) currently writes row-by-row via GenericAppenderFactory, resulting in significantly lower throughput for append-only log table tiering compared to the Paimon path.

Since Fluss uses Arrow as its internal columnar format, and both Iceberg and Parquet have native Arrow support (iceberg-arrow module, parquet-arrow module), we can bridge the Arrow batch directly to Iceberg Parquet files.

Solution

1. Make IcebergLakeWriter implement SupportsRecordBatchWrite
2. Create an Arrow batch writer that takes ArrowBatchData and writes to Iceberg Parquet files
3. Only enabled for append-only tables (same restriction as Paimon)
4. Preserve Iceberg file metrics tracking for proper table maintenance

Related: #437 (generic arrow-to-parquet issue)

Upstream Issues for True Zero-Copy Path

The current PR (#4048) implements SupportsRecordBatchWrite by reading Arrow vectors and producing GenericRecord objects. This avoids binary log deserialization but still materializes row objects because Iceberg and parquet-java lack columnar write APIs.

Upstream issues filed to enable true zero-copy Arrow-to-Parquet writes:

1. apache/parquet-java#3733 — Add ParquetWriter<VectorSchemaRoot> to the parquet-arrow module. This is the foundational primitive.
2. apache/iceberg#17748 — Add FileAppender<VectorSchemaRoot> to Iceberg's arrow module. Depends on (or works alongside) the parquet-java change.

Once these land, Fluss can replace the IcebergArrowBatchHelper (GenericRecord bridge) with a direct call to Iceberg's vectorized appender.

Anything else?

All necessary primitives exist in upstream Iceberg (iceberg-arrow 1.10.1) and Apache Parquet (parquet-arrow). No upstream changes needed for the initial implementation; upstream changes enable the optimized path.

Willingness to contribute

• I'm willing to submit a PR!

[qzyu999]

Upstream Context

The current PR (#4048) implements SupportsRecordBatchWrite by reading Arrow vectors directly and producing Iceberg GenericRecord objects. This avoids binary log deserialization but still materializes row objects because parquet-java lacks a batch/columnar write API.

For true zero-copy Arrow-to-Parquet writing (matching what Paimon achieves with paimon-arrow), the gap is in the underlying Parquet Java library:

• apache/parquet-java#3733 Filed to request batch write APIs + Arrow VectorSchemaRoot writer (Java parity with C++/Python write_table())
• apache/parquet-java#2264 Original request from 2019 (no activity)
• apache/parquet-java#3530 Active perf series that plans 'level write batching' as a future PR

Once parquet-java exposes batch write methods on ValuesWriter, the Iceberg and Fluss layers can be updated to pass Arrow column buffers directly without row materialization.