Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions website/docs/engine-spark/options.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ The following Spark configurations can be used to control read behavior for both
| `spark.sql.fluss.scan.startup.mode` | `full` | The startup mode when reading a Fluss table. Supported values: <ul><li>`full` (default): For primary key tables, reads the full snapshot and merges with log changes. For log tables, reads from the earliest offset.</li><li>`earliest`: Reads from the earliest log/changelog offset.</li><li>`latest`: Reads from the latest log/changelog offset.</li></ul>**Note:** This option only affects Structured Streaming reads, and only `latest` mode is currently supported there. Batch reads ignore it: a plain batch read is always the full table, and a time-range batch read is requested per query (see below). |
| `spark.sql.fluss.read.optimized` | `false` | If `true`, Spark will only read data from the data lake snapshot or KV snapshot, without merging log changes. This can improve read performance but may return stale data for primary key tables. |
| `spark.sql.fluss.scan.poll.timeout` | `10000ms` | The timeout for the log scanner to poll records. |
| `spark.sql.fluss.scan.maxRecordsPerPartition` | (none) | The maximum number of records per Spark input partition for batch log-table reads. Accepts a positive `Long`. When configured, each bounded Fluss bucket offset range that exceeds the value is split into multiple Spark input partitions. Disabled by default; it changes scan parallelism without changing the rows returned. |

## Per-Query Read Options

Expand Down