diff --git a/cpp/tools/skills/tsfile-cli/SKILL.md b/cpp/tools/skills/tsfile-cli/SKILL.md index 3aa94fd75..8261cc70a 100644 --- a/cpp/tools/skills/tsfile-cli/SKILL.md +++ b/cpp/tools/skills/tsfile-cli/SKILL.md @@ -1,32 +1,43 @@ - - --- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. name: tsfile-cli -description: Use when you need to inspect, preview, export, OR import an Apache TsFile (.tsfile) from the command line — list devices/tables, dump schema, read file/series metadata, count rows, sample/preview rows, or write CSV/TSV into a new .tsfile — via the project's C++ `tsfile-cli` in cpp/tools. +description: >- + Use specifically for the project's C++ `tsfile-cli` in cpp/tools: inspect, + preview, export, or sample an Apache TsFile; report metadata or per-series + counts; or use its explicit single-table CSV/TSV write command. --- # tsfile-cli -Single pipe-friendly C++ binary to inspect **and** import `.tsfile` (TsFile's analogue of -`parquet-cli`/`pqrs`). Source `cpp/tools/`. Read data → stdout, diagnostics → stderr; -`write` imports CSV/TSV → a new file. +Single pipe-friendly C++ binary to inspect `.tsfile` files and create a new +table-model TsFile from CSV/TSV (TsFile's analogue of `parquet-cli`/`pqrs`). +Source `cpp/tools/`. Read data → stdout, diagnostics → stderr. + +## Scope + +Use this skill only for the C++ `tsfile-cli` binary. Route Java +`csv2tsfile`/`parquet2tsfile`/`arrow2tsfile` and table point-count metadata +checks or backfill to the top-level `tsfile` skill when it is installed. + +The names overlap but the semantics do not: `tsfile-cli count` is a read-only +per-series report, not the Java table point-count property tool. `tsfile-cli +write` is the C++ binary's narrow one-file/stream, one-table CSV/TSV import; it +does not replace the Java batch and format-aware import tools. ## Binary diff --git a/skills/tsfile-cli/SKILL.md b/skills/tsfile-cli/SKILL.md index 37c59a207..4cbb9cc57 100644 --- a/skills/tsfile-cli/SKILL.md +++ b/skills/tsfile-cli/SKILL.md @@ -16,20 +16,29 @@ # specific language governing permissions and limitations # under the License. name: tsfile-cli -description: Use when you need to inspect, preview, export, OR import an Apache TsFile (.tsfile) from the command line — list devices/tables, dump schema, read file/series metadata, count rows, sample/preview rows, or write CSV/TSV into a new .tsfile — via the project's C++ `tsfile-cli` in cpp/tools. +description: >- + Use specifically for the project's C++ `tsfile-cli` in cpp/tools: inspect, + preview, export, or sample an Apache TsFile; report metadata or per-series + counts; or use its explicit single-table CSV/TSV write command. --- # tsfile-cli -Single pipe-friendly C++ binary to inspect **and** import `.tsfile` (TsFile's analogue of -`parquet-cli`/`pqrs`). Source `cpp/tools/`. Read data → stdout, diagnostics → stderr; -`write` imports CSV/TSV → a new file. +Single pipe-friendly C++ binary to inspect `.tsfile` files and create a new +table-model TsFile from CSV/TSV (TsFile's analogue of `parquet-cli`/`pqrs`). +Source `cpp/tools/`. Read data → stdout, diagnostics → stderr. ## Scope -Use this skill for command-line builds and operations. For Java, Python, C++, -or C SDK integration, schema design, and programmatic tree-model writes, load -the sibling `tsfile` skill at `../tsfile/SKILL.md`. +Use this skill only for the C++ `tsfile-cli` binary. For Java, Python, C++, or C +SDK integration, schema design, Java CSV/Parquet/Arrow import, Java table +point-count metadata checks or backfill, and programmatic tree-model writes, +load the sibling `tsfile` skill at `../tsfile/SKILL.md`. + +The names overlap but the semantics do not: `tsfile-cli count` is a read-only +per-series report, not the Java table point-count property tool. `tsfile-cli +write` is the C++ binary's narrow one-file/stream, one-table CSV/TSV import; it +does not replace the Java batch and format-aware import tools. ## Binary diff --git a/skills/tsfile/SKILL.md b/skills/tsfile/SKILL.md index 360b8df26..85e5f06f6 100644 --- a/skills/tsfile/SKILL.md +++ b/skills/tsfile/SKILL.md @@ -16,17 +16,25 @@ # specific language governing permissions and limitations # under the License. name: tsfile -description: Work with Apache TsFile programmatic SDKs and file-format concepts in Java, Python, C++, or C. Use for reading, writing, querying, schema or data-model design, encoding/compression decisions, performance analysis, API compatibility, and cross-language TsFile integration. Route shell inspection, preview, export, sampling, and CSV/TSV conversion to the sibling tsfile-cli skill. +description: >- + Work with the current Apache TsFile SDKs, tools, and file format. Use for + Java, Python, C++, or C reads and writes; Python DataFrame and Arrow batches; + Java CSV/Parquet/Arrow import; file properties; TAG filters; query-by-row and + metadata APIs; C++ incomplete-writer recovery and append; static/shared C++ + builds; parallel and TsBlock writes; encoding/compression factories; schema + design; performance; and cross-language integration. Route operations + specific to the C++ tsfile-cli binary to the sibling tsfile-cli skill. --- # TsFile ## Scope -Use this skill for SDK code, Tree/Table model decisions, schema design, -compatibility, and cross-language integration. Use the sibling -`../tsfile-cli/SKILL.md` for shell-oriented inspection, preview, export, -sampling, and CSV/TSV-to-TsFile conversion. +Use this skill for SDK code, Java import and point-count tools, Tree/Table model +decisions, schema design, compatibility, and cross-language integration. Use +the sibling `../tsfile-cli/SKILL.md` for the C++ `tsfile-cli` binary: file +inspection, preview, export, sampling, and its explicit single-table CSV/TSV +`write` operation. ## Operating Rules @@ -56,6 +64,19 @@ Read only the files required by the current task: - Model selection, schema, data types, and generic read/write workflow: `references/core-concepts.md` - Java SDK code and API guardrails: `references/java.md` +- Java CSV/Parquet/Arrow import and table point-count tools: + `references/java-tools.md` +- Python DataFrame datasets, pandas conversion, and Arrow batch I/O: + `references/dataframe-arrow.md` +- File-level properties and Table-model TAG filters across bindings: + `references/properties-filters.md` +- C++ static/shared linking and parallel write configuration: + `references/cpp-build-write.md` +- Current Java and C++ encoding/compression factory matrices: + `references/encoding-compression.md` +- Query-by-row, timeseries metadata, C++ incomplete-file recovery and append, + Java TsBlock writes, custom/buffered input, and Tablet object/size APIs: + `references/advanced-apis.md` - Python SDK code and binding-specific behavior: `references/python.md` - C++ SDK code and resource management: `references/cpp.md` - C wrapper entry points and lifecycle: `references/c.md` @@ -80,6 +101,38 @@ For compatibility questions, report the local source version and the requested release separately. Never silently combine signatures from different versions or language bindings. +## Current Baseline Routing + +- Use `references/dataframe-arrow.md` for `TsFileDataFrame`, pandas conversion, + PyArrow `Table`/`RecordBatch`, or Arrow C Data batch paths. +- Use `references/properties-filters.md` for binary file properties or + Table-model TAG predicate construction and composition. +- Use `references/cpp-build-write.md` for C++ library linkage or write + concurrency. Configure the current source APIs; do not emit legacy aliases. +- Use `references/encoding-compression.md` before naming a factory or claiming + that an encoding/type or compressor combination is constructible. +- Use `references/advanced-apis.md` for C++ crash recovery and append through + `RestorableTsFileIOWriter`, plus the remaining current advanced read and write + surfaces listed in Offline Reference Routing. + +Treat these as baseline capabilities. Do not add alternate calls, degraded +results, or compatibility branches for earlier TsFile source lines unless the +user explicitly targets a different checkout or release. + +## Java Tools + +- Route `csv2tsfile`, `parquet2tsfile`, `arrow2tsfile`, schema-driven import, + import auto-inference, directory import, and failed-file handling to the Java + tools in `java/tools/`. Read `references/java-tools.md` before constructing or + running a command. +- Route requests to inspect or backfill the persisted table-level point-count + property to the Java `tsfile-table-point-count` tool. This operation can + replace the input file when metadata is missing, so follow the mutation + checks in `references/java-tools.md`. +- Do not substitute C++ `tsfile-cli count` for the Java point-count tool. The + former reports per-series counts without changing the file; the latter + validates and may backfill table-level metadata. + ## Bundled Resources - Run `scripts/resolve-version.sh [--root ]` to obtain Maven, C++, diff --git a/skills/tsfile/references/advanced-apis.md b/skills/tsfile/references/advanced-apis.md new file mode 100644 index 000000000..c8672657f --- /dev/null +++ b/skills/tsfile/references/advanced-apis.md @@ -0,0 +1,99 @@ + + +# Current Advanced APIs + +Load only the section matching the task and verify exact constructors against +the listed current source. + +## Row-window Queries + +- C++ exposes `TsFileReader::queryByRow` for Tree paths and for Table columns, + with `offset`, `limit`, optional Table TAG filter, and optional batch size. + `TsFileTreeReader::queryByRow` accepts device and measurement lists. +- Python exposes `query_tree_by_row` and `query_table_by_row`; Table queries + accept `tag_filter` and `batch_size`. +- Treat `limit < 0` as unlimited and reject negative offsets. Close or destroy + every result set through the binding that created it. + +## Timeseries Metadata + +- C++ `TsFileReader::get_timeseries_metadata()` returns metadata for all + devices; its device-list overload restricts the request. +- Python `get_timeseries_metadata(device_ids=None)` returns a mapping keyed by + the full device-segment tuple. `None` selects all devices, while `[]` returns + an empty map; null TAG segments remain `None`. +- C callers must use the all-devices or selected-devices declarations and the + matching free routine in `tsfile_cwrapper.h`. + +## Java TsBlock Write + +Use `TableTsBlock2TsFileWriter` when the input is already a Table-model +`TsBlock`. Its constructor maps time, TAG, and FIELD column indexes and can +generate a monotonically increasing time column per device. Feed complete +blocks through `write(TsBlock)`, then close the writer. Build the mapping from +the current `TableSchema`; do not infer constructor positions from memory. + +## Java Tablet Object and Size APIs + +- Use `Tablet.addObjectPathValue` only for an `OBJECT` column; overloads accept + a measurement name or column index and a `String` or `byte[]` path. +- Use `Tablet.serializedSize()` to allocate or validate the serialized Tablet + payload. Do not replace it with a RAM-size estimate. + +## Custom and Buffered Java Input + +`TsFileSequenceReader` accepts a `TsFileInput`, enabling custom storage/input +implementations. For local buffered reads, construct `BufferedTsFileInput(Path, +bufferSize)` and pass it to the reader. Buffer size must be positive, and the +reader owns and closes the supplied input if initialization fails or when the +reader is closed. + +## C++ Recovery and Append + +Use `storage::RestorableTsFileIOWriter` to inspect and optionally truncate an +incomplete file before continuing a write. After `open(path, true)`, require +`has_crashed()` and `can_write()` before constructing `TsFileTreeWriter` or +`TsFileTableWriter` with the restorable writer. The recovered schema is used; +do not register a replacement schema. A complete file reports +`can_write() == false`. Keep the restorable writer alive longer than the facade +writer and preserve increasing timestamps beyond the recovered last values. + +## Java BitMap Ranges + +Use `markRange(start, length)` and `unmarkRange(start, length)` for mutations, +and `isRangeAnyMarked`, `isRangeAllMarked`, or `isRangeNoneMarked` for range +tests. Ranges are `(start, length)`, not `(start, end)`; empty ranges return +`false`, `true`, and `true`, respectively. Out-of-bounds ranges throw. + +## Source Anchors + +- Row queries: `cpp/src/reader/tsfile_reader.h`, + `cpp/src/reader/tsfile_tree_reader.h`, and `python/tsfile/tsfile_reader.pyx` +- Metadata: those readers, `cpp/src/cwrapper/tsfile_cwrapper.h`, + `python/tsfile/schema.py`, and `python/tests/test_reader_metadata.py` +- TsBlock: `java/tsfile/src/main/java/org/apache/tsfile/write/v4/TableTsBlock2TsFileWriter.java` +- Tablet: `java/tsfile/src/main/java/org/apache/tsfile/write/record/Tablet.java` +- Input: `TsFileSequenceReader.java`, `TsFileInput.java`, and + `BufferedTsFileInput.java` under + `java/tsfile/src/main/java/org/apache/tsfile/read/reader/` +- Recovery: `cpp/src/file/restorable_tsfile_io_writer.h` and the current C++ + Tree/Table writer headers and recovery tests +- BitMap: `java/common/src/main/java/org/apache/tsfile/utils/BitMap.java` and + `java/tsfile/src/test/java/org/apache/tsfile/utils/BitMapTest.java` diff --git a/skills/tsfile/references/cpp-build-write.md b/skills/tsfile/references/cpp-build-write.md new file mode 100644 index 000000000..87d8d7307 --- /dev/null +++ b/skills/tsfile/references/cpp-build-write.md @@ -0,0 +1,83 @@ + + +# C++ Build and Parallel Write + +## Shared or Static libtsfile + +`TSFILE_BUILD_SHARED` selects the library kind and defaults to `ON`. + +```sh +# Maven profile: shared default or static override +./mvnw clean verify -P with-cpp +./mvnw clean verify -P with-cpp -Dtsfile.build.shared=OFF + +# Direct CMake static build +cmake -S cpp -B cpp/build/static \ + -DTSFILE_BUILD_SHARED=OFF -DBUILD_TEST=OFF +cmake --build cpp/build/static --target tsfile + +# Repository wrapper +cd cpp && bash build.sh --build-static +``` + +Use a separate build directory when switching library kind. A direct MSVC +consumer of the static archive must define `TSFILE_STATIC`; consumers of the +exported CMake target inherit the required definition. Verify the artifact in +`cpp/target/build/lib` for Maven or the selected CMake build tree. + +`ENABLE_SIMD` also defaults to `ON` in the current CMake and Maven builds. Set +it explicitly only for portability diagnosis or controlled benchmarking; it +is independent of static/shared selection and thread support. + +## Parallel Write + +Compile thread support with `ENABLE_THREADS=ON` (the default). `OFF` strips the +threaded paths. Configure the current global worker pool through +`common/global.h`: + +```cpp +#include "common/global.h" + +common::set_parallel_write_enabled(true); +int status = common::set_thread_count(4); // valid range: 1..64 +``` + +The current baseline uses `common::set_parallel_write_enabled` and +`common::set_thread_count`; use those exact names and namespace. The pool is +shared by parallel read and write paths. Configure it before creating active +readers/writers. Changing the count after initialization rebuilds the pool and +must never race with an operation using it. + +Current source initializes parallel writing as enabled and the pool size as 6. +Do not substitute a hardware-concurrency default from older prose. Parallel +work is used only when the compiled path and runtime conditions qualify; keep +the tablet/batch write contract correct when it falls back to serial work. + +Always check write return codes. If a multi-column batch fails partway, stop +using that writer rather than attempting to flush a potentially misaligned +partial batch. + +## Source Anchors + +- Link selection: `cpp/CMakeLists.txt`, `cpp/src/CMakeLists.txt`, + `cpp/build.sh`, `cpp/pom.xml`, and `cpp/README.md` +- Runtime API and defaults: `cpp/src/common/global.h`, + `cpp/src/common/global.cc`, and `cpp/src/common/config/config.h` +- Write behavior: `cpp/src/writer/tsfile_writer.cc` and current writer tests diff --git a/skills/tsfile/references/dataframe-arrow.md b/skills/tsfile/references/dataframe-arrow.md new file mode 100644 index 000000000..843c00def --- /dev/null +++ b/skills/tsfile/references/dataframe-arrow.md @@ -0,0 +1,100 @@ + + +# Python DataFrame and Arrow + +Use the current Python binding and its tests as the API authority. + +## Select the Surface + +- Use `to_dataframe` for an eager pandas result, or batched pandas iterator, + from one Tree- or Table-model file. +- Use `dataframe_to_tsfile` for an inferred Table-model schema and a complete + pandas-to-TsFile conversion. +- Use `TsFileDataFrame` for a lazy unified numeric view across one or more + TsFile shards, metadata selection, or timestamp-aligned series access. +- Use Arrow batches for columnar interoperability without Python row loops. +- Use `references/java-tools.md` instead for Arrow file import through the Java + `arrow2tsfile` command. + +## pandas Conversion + +```python +from tsfile import dataframe_to_tsfile, to_dataframe + +frame = to_dataframe( + "input.tsfile", table_name="sensors", column_names=["device", "temp"], + start_time=0, end_time=10_000, max_row_num=100_000, +) +dataframe_to_tsfile( + frame, "output.tsfile", table_name="sensors", + time_column="time", tag_column=["device"], +) +``` + +Set `as_iterator=True` on `to_dataframe` to consume pandas batches. The writer +lowercases column names, requires an integer time column when one is supplied, +and otherwise uses a `time` column or the DataFrame index. Validate inferred +object-column types before writing production data. + +## Lazy Multi-file Dataset + +```python +from tsfile import TsFileDataFrame + +with TsFileDataFrame(["part-1.tsfile", "part-2.tsfile"], show_progress=False) as ds: + metadata = ds.list_timeseries_metadata() + selected = ds[ds["count"] > 0] + aligned = selected.loc[0:10_000, [0, 1]] +``` + +A single dataset must not mix Tree- and Table-model files. The root +`TsFileDataFrame` owns readers; use its context manager and do not expect a +subset view to close shared readers. Boolean dataset selection is positional: +build the mask from `ds[...]`, as above; use `list_timeseries_metadata()` for +named inspection rather than as the mask source. + +## Arrow Batch I/O + +```python +with reader.query_table( + "sensors", ["device", "temp"], batch_size=8192 +) as result: + while True: + table = result.read_arrow_batch() + if table is None: + break + consume(table) # pyarrow.Table + +with TsFileTableWriter("output.tsfile", schema) as writer: + writer.write_arrow_batch(record_batch_or_table) +``` + +Use `batch_size > 0` for Arrow reads. Arrow writes accept a +`pyarrow.RecordBatch` or `pyarrow.Table`; include the registered time column and +match all remaining columns to the Table schema. Use the wrapper writer when +possible so it resolves the time-column index from the schema. + +## Source Anchors + +- `python/tsfile/utils.py`, `python/tsfile/dataset/`, and the current Python + reader/writer bindings +- `python/tsfile/tsfile_table_writer.py` +- `python/tests/test_dataframe.py`, `python/tests/test_tsfile_dataset.py`, + `python/tests/test_batch_arrow.py`, and `python/tests/test_write_arrow.py` diff --git a/skills/tsfile/references/docs-map.yaml b/skills/tsfile/references/docs-map.yaml index 4fb4f342f..ed6e21305 100644 --- a/skills/tsfile/references/docs-map.yaml +++ b/skills/tsfile/references/docs-map.yaml @@ -113,8 +113,73 @@ dataframe: guide_zh: https://tsfile.apache.org/zh/UserGuide/latest/DataFrame/TsFileDataFrame.html version_scope: v2.x_latest_alias local_authorities: - - python/tsfile/ - - python/tests/ + - python/tsfile/utils.py + - python/tsfile/dataset/ + - python/tsfile/tsfile_reader.pyx + - python/tsfile/tsfile_writer.pyx + - python/tsfile/tsfile_table_writer.py + - python/tests/test_dataframe.py + - python/tests/test_tsfile_dataset.py + - python/tests/test_batch_arrow.py + - python/tests/test_write_arrow.py + - skills/tsfile/references/dataframe-arrow.md + +current_capabilities: + file_properties_and_tag_filters: + local_authorities: + - java/tsfile/src/main/java/org/apache/tsfile/read/filter/factory/TagFilterBuilder.java + - java/tsfile/src/main/java/org/apache/tsfile/read/v4/ITsFileReader.java + - java/tsfile/src/main/java/org/apache/tsfile/write/v4/ITsFileWriter.java + - cpp/src/reader/filter/tag_filter.h + - cpp/src/reader/tsfile_reader.h + - cpp/src/writer/tsfile_writer.h + - cpp/src/writer/tsfile_table_writer.h + - cpp/src/cwrapper/tsfile_cwrapper.h + - python/tsfile/tag_filter.py + - python/tsfile/tsfile_reader.pyx + - python/tsfile/tsfile_writer.pyx + - python/tsfile/tsfile_table_writer.py + - skills/tsfile/references/properties-filters.md + cpp_build_and_parallel_write: + local_authorities: + - cpp/CMakeLists.txt + - cpp/src/CMakeLists.txt + - cpp/build.sh + - cpp/src/common/global.h + - cpp/src/common/global.cc + - cpp/src/common/config/config.h + - cpp/src/writer/tsfile_writer.cc + - skills/tsfile/references/cpp-build-write.md + encoding_and_compression_factories: + local_authorities: + - java/tsfile/src/main/java/org/apache/tsfile/file/metadata/enums/TSEncoding.java + - java/tsfile/src/main/java/org/apache/tsfile/encoding/encoder/TSEncodingBuilder.java + - java/tsfile/src/main/java/org/apache/tsfile/encoding/decoder/Decoder.java + - java/tsfile/src/main/java/org/apache/tsfile/file/metadata/enums/CompressionType.java + - java/tsfile/src/main/java/org/apache/tsfile/compress/ICompressor.java + - java/tsfile/src/main/java/org/apache/tsfile/compress/IUnCompressor.java + - cpp/src/encoding/encoder_factory.h + - cpp/src/encoding/decoder_factory.h + - cpp/src/compress/compressor_factory.h + - cpp/src/common/db_common.h + - skills/tsfile/references/encoding-compression.md + advanced_apis: + local_authorities: + - cpp/src/reader/tsfile_reader.h + - cpp/src/reader/tsfile_tree_reader.h + - cpp/src/cwrapper/tsfile_cwrapper.h + - python/tsfile/tsfile_reader.pyx + - python/tsfile/schema.py + - java/tsfile/src/main/java/org/apache/tsfile/write/v4/TableTsBlock2TsFileWriter.java + - java/tsfile/src/main/java/org/apache/tsfile/write/record/Tablet.java + - java/tsfile/src/main/java/org/apache/tsfile/read/TsFileSequenceReader.java + - java/tsfile/src/main/java/org/apache/tsfile/read/reader/TsFileInput.java + - java/tsfile/src/main/java/org/apache/tsfile/read/reader/BufferedTsFileInput.java + - cpp/src/file/restorable_tsfile_io_writer.h + - cpp/src/writer/tsfile_tree_writer.h + - cpp/src/writer/tsfile_table_writer.h + - java/common/src/main/java/org/apache/tsfile/utils/BitMap.java + - skills/tsfile/references/advanced-apis.md ecosystem: version_policy: verify_connector_dependencies_before_use @@ -123,11 +188,33 @@ ecosystem: spark: https://tsfile.apache.org/UserGuide/latest/Ecosystem-Integration/Spark-TsFile.html tools: - owner_skill: - path: ../tsfile-cli/SKILL.md - base: skill_root - cli: https://tsfile.apache.org/UserGuide/latest/Tools/Tsfile-CLI.html - viewer: https://tsfile.apache.org/UserGuide/latest/Tools/Tsfile-Viewer.html - local_authorities: - - cpp/tools/README.md - - cpp/tools/ + java: + owner_skill: + path: SKILL.md + base: skill_root + import: + formats: + - csv + - parquet + - arrow + local_authorities: + - java/tools/README.md + - java/tools/src/main/java/org/apache/tsfile/tools/ + - java/tools/src/assembly/resources/tools/ + - skills/tsfile/references/java-tools.md + table_point_count: + local_authorities: + - java/tools/README.md + - java/tsfile/src/main/java/org/apache/tsfile/utils/TsFileTablePointCountTool.java + - java/tools/src/assembly/resources/tools/tsfile-table-point-count.sh + - java/tools/src/assembly/resources/tools/tsfile-table-point-count.bat + - skills/tsfile/references/java-tools.md + cpp_cli: + owner_skill: + path: ../tsfile-cli/SKILL.md + base: skill_root + cli: https://tsfile.apache.org/UserGuide/latest/Tools/Tsfile-CLI.html + viewer: https://tsfile.apache.org/UserGuide/latest/Tools/Tsfile-Viewer.html + local_authorities: + - cpp/tools/README.md + - cpp/tools/ diff --git a/skills/tsfile/references/encoding-compression.md b/skills/tsfile/references/encoding-compression.md new file mode 100644 index 000000000..b745687ca --- /dev/null +++ b/skills/tsfile/references/encoding-compression.md @@ -0,0 +1,77 @@ + + +# Current Encoding and Compression Factories + +Prefer schema constructors for normal file writes. Use these low-level +factories only for codec, page, or format work. An enum member does not prove +that a factory can construct the requested codec. + +## Java + +Validate with `TSEncoding.isSupported(dataType, encoding)`, create encoders with +`TSEncodingBuilder.getEncodingBuilder(encoding).getEncoder(dataType)`, and +create decoders with `Decoder.getDecoderByType(encoding, dataType)`. + +| data type | current supported encodings | +|---|---| +| `BOOLEAN` | `PLAIN`, `RLE` | +| `INT32`, `INT64`, `DATE`, `TIMESTAMP` | `PLAIN`, `RLE`, `TS_2DIFF`, `GORILLA`, `ZIGZAG`, `CHIMP`, `SPRINTZ`, `RLBE` | +| `FLOAT` | `PLAIN`, `RLE`, `TS_2DIFF`, `GORILLA_V1`, `GORILLA`, `CHIMP`, `SPRINTZ`, `RLBE` | +| `DOUBLE` | the FLOAT set plus `CAMEL` | +| `TEXT`, `STRING` | `PLAIN`, `DICTIONARY` | +| `BLOB`, `OBJECT` | `PLAIN` | + +Create compressors with `ICompressor.getCompressor(CompressionType)` and +decompressors with `IUnCompressor.getUnCompressor(CompressionType)`. The +current Java factory set is `UNCOMPRESSED`, `SNAPPY`, `GZIP`, `LZ4`, `ZSTD`, +and `LZMA2`. + +## C++ + +Use `EncoderFactory::alloc_time_encoder`, +`EncoderFactory::alloc_value_encoder`, `DecoderFactory::alloc_time_decoder`, +`DecoderFactory::alloc_value_decoder`, and each factory's `free` function. +Current time factories accept `PLAIN` or `TS_2DIFF`. + +| value encoding | current accepted data types | +|---|---| +| `PLAIN` | all types handled by `PlainEncoder`/`PlainDecoder` | +| `DICTIONARY` | `STRING`, `TEXT` | +| `RLE` | `INT32`, `DATE`, `INT64`, `TIMESTAMP` | +| `TS_2DIFF`, `GORILLA` | `INT32`, `DATE`, `INT64`, `TIMESTAMP`, `FLOAT`, `DOUBLE` | +| `ZIGZAG` | `INT32`, `INT64` | +| `SPRINTZ` | `INT32`, `INT64`, `FLOAT`, `DOUBLE` | + +`CompressorFactory::alloc_compressor` always supports `UNCOMPRESSED`; `SNAPPY`, +`GZIP`, `LZO`, and `LZ4` require their corresponding compile definitions. +`SDT`, `PAA`, and `PLA` are enum values but the current compressor factory +returns no implementation for them. Check for `nullptr` and release successful +allocations with `CompressorFactory::free`. + +Python exposes schema encoding/compression enums through the native binding but +does not expose these Java/C++ low-level factories as an equivalent Python API. + +## Source Anchors + +- Java: `TSEncoding.java`, `TSEncodingBuilder.java`, `Decoder.java`, + `CompressionType.java`, `ICompressor.java`, and `IUnCompressor.java` +- C++: `cpp/src/encoding/encoder_factory.h`, + `cpp/src/encoding/decoder_factory.h`, + `cpp/src/compress/compressor_factory.h`, and `cpp/src/common/db_common.h` diff --git a/skills/tsfile/references/java-tools.md b/skills/tsfile/references/java-tools.md new file mode 100644 index 000000000..e344226c6 --- /dev/null +++ b/skills/tsfile/references/java-tools.md @@ -0,0 +1,89 @@ + + +# Java Tools + +Use this reference for the packaged Java command-line tools. Use +`java/tools/README.md`, its tests, and the current source as the exact command +authority. Do not infer these commands from the C++ `tsfile-cli` interface. + +## Locate or Build + +- In a packaged distribution, run scripts from `$TSFILE_HOME/tools/`; they use + the jars under `$TSFILE_HOME/lib/`. +- In a source checkout, build the Java distribution with the checkout's + documented `./mvnw clean package -P with-java -DskipTests` flow before using + the packaged scripts. Treat `java/tools/src/assembly/resources/tools/` as + packaging source, not as an independently installed runtime. +- Require Java and `JAVA_HOME` as documented by the current checkout. + +## Import CSV, Parquet, or Arrow + +Prefer the Java importer for format-aware or batch conversion: a source file or +directory, CSV/Parquet/Arrow input, explicit schema mapping, schema inference, +failed-file collection, threading, or CSV chunking. + +```sh +$TSFILE_HOME/tools/csv2tsfile.sh --source input.csv --target output +$TSFILE_HOME/tools/parquet2tsfile.sh --source parquet-dir --target output +$TSFILE_HOME/tools/arrow2tsfile.sh --source input.arrow --target output +$TSFILE_HOME/tools/csv2tsfile.sh --source input-dir --target output \ + --fail_dir failed --schema import.schema +``` + +Use the corresponding `.bat` scripts on Windows. Omit `--schema` only when the +current auto-inference rules are acceptable. Read `java/tools/README.md` before +generating a schema or relying on time-column, tag-column, null, delimiter, +type-inference, file-naming, or failure-handling behavior. + +Use the C++ `tsfile-cli write` command only when the requested operation is its +narrow pipe-friendly case: one CSV/TSV stream or file, one new table-model +TsFile, and an explicit inline `--columns` schema. + +## Check or Backfill Table Point Count + +Use the Java point-count tool for the persisted table-level property: + +```sh +$TSFILE_HOME/tools/tsfile-table-point-count.sh /data/example.tsfile +``` + +Use the `.bat` wrapper on Windows. Pass exactly one complete TsFile. The count +is the total number of non-null FIELD values for each table; TAG and time +columns are excluded. + +Interpret `UPDATED` as an in-place metadata backfill, `ALREADY_PRESENT` as no +change, and `NO_TABLE` as a tree-only/no-table result. Before running it, +resolve the exact input path, ensure the containing directory is writable and +has room for a complete temporary copy, and preserve a backup when the data is +not otherwise recoverable. Missing and incomplete files must remain errors. + +Do not use C++ `tsfile-cli count` as a substitute. That command reads +per-series statistics and prints counts; it neither validates nor writes the +table-level point-count property. + +## Source Anchors + +- Import behavior: `java/tools/README.md`, + `java/tools/src/main/java/org/apache/tsfile/tools/`, and its tests. +- Import wrappers: `java/tools/src/assembly/resources/tools/`. +- Point-count implementation: + `java/tsfile/src/main/java/org/apache/tsfile/utils/TsFileTablePointCountTool.java`. +- Point-count wrapper and user contract: `java/tools/README.md` and + `java/tools/src/assembly/resources/tools/tsfile-table-point-count.*`. diff --git a/skills/tsfile/references/performance.md b/skills/tsfile/references/performance.md index 6972f0aad..6a745844e 100644 --- a/skills/tsfile/references/performance.md +++ b/skills/tsfile/references/performance.md @@ -44,6 +44,8 @@ than a universal setting. ## Encoding and Compression +- Read `references/encoding-compression.md` for the current constructible + factory/type matrix. Do not infer factory support from enum membership. - Start from the implementation defaults for the target version and language. - Match encodings to observed data distribution only after measuring: monotonic integers, floating-point continuity, boolean/cardinality patterns, and text diff --git a/skills/tsfile/references/properties-filters.md b/skills/tsfile/references/properties-filters.md new file mode 100644 index 000000000..6fd461d24 --- /dev/null +++ b/skills/tsfile/references/properties-filters.md @@ -0,0 +1,80 @@ + + +# File Properties and TAG Filters + +## File-level Properties + +Treat property values as untyped binary application metadata. Add or replace +them only while a writer is open, close the writer to persist them, and define +an explicit portable encoding for numbers or structures. + +- Java: call `ITsFileWriter.addTsFileProperty(String, byte[])` and + `ITsFileReader.getTsFileProperties()`. +- Python: call `add_tsfile_property(str, bytes)` and + `get_tsfile_properties()`. The setter accepts exactly `bytes`; the reader + preserves null as `None` and a non-null empty value as `b""`. +- C++: call `add_tsfile_property(...)` on `TsFileWriter` or + `TsFileTableWriter`, and `TsFileReader::get_tsfile_properties()`. The + `TsFileProperties` value distinguishes null from an empty byte vector. +- C: use the declarations and matching free function in + `cpp/src/cwrapper/tsfile_cwrapper.h`; do not guess buffer ownership. + +The Java table point-count metadata is a reserved use of file properties. Use +`references/java-tools.md` for inspection or backfill instead of editing those +keys directly. + +## Table TAG Filters + +Build TAG predicates against a `TableSchema`, apply them only to Table-model +queries, and pass field/time restrictions through their separate query +arguments. TAG values are strings; do not apply a TAG builder to FIELD columns. + +- Java `TagFilterBuilder`: `eq`, `neq`, `lt`, `lteq`, `gt`, `gteq`, + `betweenAnd`, `notBetweenAnd`, `regExp`, `notRegExp`, `like`, `notLike`, plus + `and`, `or`, and `not`. Pass the result to the five-argument + `ITsFileReader.query(..., Filter tagFilter)` overload. +- Python factories: `tag_eq`, `tag_neq`, `tag_lt`, `tag_lteq`, `tag_gt`, + `tag_gteq`, `tag_regexp`, `tag_not_regexp`, `tag_is_null`, + `tag_is_not_null`, `tag_between`, and `tag_not_between`. Compose with `&`, + `|`, and `~`, then pass `tag_filter=` to `query_table` or + `query_table_by_row`. +- C++ `TagFilterBuilder`: use `eq`/`neq`/comparison, regex, null, range, and + static logical builders from `cpp/src/reader/filter/tag_filter.h`; pass the + resulting `Filter*` to the matching `TsFileReader::query` or `queryByRow` + overload and follow the current ownership implementation. +- C wrapper: construct `TagFilterHandle` values with the declared factory and + composition functions, pass them to the tag-filter query entry point, and + release them with `tsfile_tag_filter_free`. + +Do not translate operator names mechanically across bindings: Java additionally +exposes LIKE, while current C++/Python expose explicit null predicates. + +## Source Anchors + +- Java: `java/tsfile/src/main/java/org/apache/tsfile/read/filter/factory/TagFilterBuilder.java`, + `java/tsfile/src/main/java/org/apache/tsfile/read/v4/ITsFileReader.java`, and + `java/examples/src/main/java/org/apache/tsfile/v4/TagFilterExample.java` +- Python: `python/tsfile/tag_filter.py`, `python/tsfile/tsfile_reader.pyx`, and + `python/tests/test_tag_filter*.py` +- C/C++: `cpp/src/reader/filter/tag_filter.h`, + `cpp/src/reader/tsfile_reader.h`, and `cpp/src/cwrapper/tsfile_cwrapper.h` +- Properties: the current writer/reader interfaces plus + `python/tests/test_tsfile_properties.py` and + `cpp/test/writer/tsfile_properties_test.cc`