Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 34 additions & 23 deletions cpp/tools/skills/tsfile-cli/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,43 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

---
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
name: tsfile-cli
description: Use when you need to inspect, preview, export, OR import an Apache TsFile (.tsfile) from the command line — list devices/tables, dump schema, read file/series metadata, count rows, sample/preview rows, or write CSV/TSV into a new .tsfile — via the project's C++ `tsfile-cli` in cpp/tools.
description: >-
Use specifically for the project's C++ `tsfile-cli` in cpp/tools: inspect,
preview, export, or sample an Apache TsFile; report metadata or per-series
counts; or use its explicit single-table CSV/TSV write command.
---

# tsfile-cli

Single pipe-friendly C++ binary to inspect **and** import `.tsfile` (TsFile's analogue of
`parquet-cli`/`pqrs`). Source `cpp/tools/`. Read data → stdout, diagnostics → stderr;
`write` imports CSV/TSV → a new file.
Single pipe-friendly C++ binary to inspect `.tsfile` files and create a new
table-model TsFile from CSV/TSV (TsFile's analogue of `parquet-cli`/`pqrs`).
Source `cpp/tools/`. Read data → stdout, diagnostics → stderr.

## Scope

Use this skill only for the C++ `tsfile-cli` binary. Route Java
`csv2tsfile`/`parquet2tsfile`/`arrow2tsfile` and table point-count metadata
checks or backfill to the top-level `tsfile` skill when it is installed.

The names overlap but the semantics do not: `tsfile-cli count` is a read-only
per-series report, not the Java table point-count property tool. `tsfile-cli
write` is the C++ binary's narrow one-file/stream, one-table CSV/TSV import; it
does not replace the Java batch and format-aware import tools.

## Binary

Expand Down
23 changes: 16 additions & 7 deletions skills/tsfile-cli/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,20 +16,29 @@
# specific language governing permissions and limitations
# under the License.
name: tsfile-cli
description: Use when you need to inspect, preview, export, OR import an Apache TsFile (.tsfile) from the command line — list devices/tables, dump schema, read file/series metadata, count rows, sample/preview rows, or write CSV/TSV into a new .tsfile — via the project's C++ `tsfile-cli` in cpp/tools.
description: >-
Use specifically for the project's C++ `tsfile-cli` in cpp/tools: inspect,
preview, export, or sample an Apache TsFile; report metadata or per-series
counts; or use its explicit single-table CSV/TSV write command.
---

# tsfile-cli

Single pipe-friendly C++ binary to inspect **and** import `.tsfile` (TsFile's analogue of
`parquet-cli`/`pqrs`). Source `cpp/tools/`. Read data → stdout, diagnostics → stderr;
`write` imports CSV/TSVa new file.
Single pipe-friendly C++ binary to inspect `.tsfile` files and create a new
table-model TsFile from CSV/TSV (TsFile's analogue of `parquet-cli`/`pqrs`).
Source `cpp/tools/`. Read datastdout, diagnostics → stderr.

## Scope

Use this skill for command-line builds and operations. For Java, Python, C++,
or C SDK integration, schema design, and programmatic tree-model writes, load
the sibling `tsfile` skill at `../tsfile/SKILL.md`.
Use this skill only for the C++ `tsfile-cli` binary. For Java, Python, C++, or C
SDK integration, schema design, Java CSV/Parquet/Arrow import, Java table
point-count metadata checks or backfill, and programmatic tree-model writes,
load the sibling `tsfile` skill at `../tsfile/SKILL.md`.

The names overlap but the semantics do not: `tsfile-cli count` is a read-only
per-series report, not the Java table point-count property tool. `tsfile-cli
write` is the C++ binary's narrow one-file/stream, one-table CSV/TSV import; it
does not replace the Java batch and format-aware import tools.

## Binary

Expand Down
63 changes: 58 additions & 5 deletions skills/tsfile/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,17 +16,25 @@
# specific language governing permissions and limitations
# under the License.
name: tsfile
description: Work with Apache TsFile programmatic SDKs and file-format concepts in Java, Python, C++, or C. Use for reading, writing, querying, schema or data-model design, encoding/compression decisions, performance analysis, API compatibility, and cross-language TsFile integration. Route shell inspection, preview, export, sampling, and CSV/TSV conversion to the sibling tsfile-cli skill.
description: >-
Work with the current Apache TsFile SDKs, tools, and file format. Use for
Java, Python, C++, or C reads and writes; Python DataFrame and Arrow batches;
Java CSV/Parquet/Arrow import; file properties; TAG filters; query-by-row and
metadata APIs; C++ incomplete-writer recovery and append; static/shared C++
builds; parallel and TsBlock writes; encoding/compression factories; schema
design; performance; and cross-language integration. Route operations
specific to the C++ tsfile-cli binary to the sibling tsfile-cli skill.
---

# TsFile

## Scope

Use this skill for SDK code, Tree/Table model decisions, schema design,
compatibility, and cross-language integration. Use the sibling
`../tsfile-cli/SKILL.md` for shell-oriented inspection, preview, export,
sampling, and CSV/TSV-to-TsFile conversion.
Use this skill for SDK code, Java import and point-count tools, Tree/Table model
decisions, schema design, compatibility, and cross-language integration. Use
the sibling `../tsfile-cli/SKILL.md` for the C++ `tsfile-cli` binary: file
inspection, preview, export, sampling, and its explicit single-table CSV/TSV
`write` operation.

## Operating Rules

Expand Down Expand Up @@ -56,6 +64,19 @@ Read only the files required by the current task:
- Model selection, schema, data types, and generic read/write workflow:
`references/core-concepts.md`
- Java SDK code and API guardrails: `references/java.md`
- Java CSV/Parquet/Arrow import and table point-count tools:
`references/java-tools.md`
- Python DataFrame datasets, pandas conversion, and Arrow batch I/O:
`references/dataframe-arrow.md`
- File-level properties and Table-model TAG filters across bindings:
`references/properties-filters.md`
- C++ static/shared linking and parallel write configuration:
`references/cpp-build-write.md`
- Current Java and C++ encoding/compression factory matrices:
`references/encoding-compression.md`
- Query-by-row, timeseries metadata, C++ incomplete-file recovery and append,
Java TsBlock writes, custom/buffered input, and Tablet object/size APIs:
`references/advanced-apis.md`
- Python SDK code and binding-specific behavior: `references/python.md`
- C++ SDK code and resource management: `references/cpp.md`
- C wrapper entry points and lifecycle: `references/c.md`
Expand All @@ -80,6 +101,38 @@ For compatibility questions, report the local source version and the requested
release separately. Never silently combine signatures from different versions
or language bindings.

## Current Baseline Routing

- Use `references/dataframe-arrow.md` for `TsFileDataFrame`, pandas conversion,
PyArrow `Table`/`RecordBatch`, or Arrow C Data batch paths.
- Use `references/properties-filters.md` for binary file properties or
Table-model TAG predicate construction and composition.
- Use `references/cpp-build-write.md` for C++ library linkage or write
concurrency. Configure the current source APIs; do not emit legacy aliases.
- Use `references/encoding-compression.md` before naming a factory or claiming
that an encoding/type or compressor combination is constructible.
- Use `references/advanced-apis.md` for C++ crash recovery and append through
`RestorableTsFileIOWriter`, plus the remaining current advanced read and write
surfaces listed in Offline Reference Routing.

Treat these as baseline capabilities. Do not add alternate calls, degraded
results, or compatibility branches for earlier TsFile source lines unless the
user explicitly targets a different checkout or release.

## Java Tools

- Route `csv2tsfile`, `parquet2tsfile`, `arrow2tsfile`, schema-driven import,
import auto-inference, directory import, and failed-file handling to the Java
tools in `java/tools/`. Read `references/java-tools.md` before constructing or
running a command.
- Route requests to inspect or backfill the persisted table-level point-count
property to the Java `tsfile-table-point-count` tool. This operation can
replace the input file when metadata is missing, so follow the mutation
checks in `references/java-tools.md`.
- Do not substitute C++ `tsfile-cli count` for the Java point-count tool. The
former reports per-series counts without changing the file; the latter
validates and may backfill table-level metadata.

## Bundled Resources

- Run `scripts/resolve-version.sh [--root <checkout>]` to obtain Maven, C++,
Expand Down
99 changes: 99 additions & 0 deletions skills/tsfile/references/advanced-apis.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Current Advanced APIs

Load only the section matching the task and verify exact constructors against
the listed current source.

## Row-window Queries

- C++ exposes `TsFileReader::queryByRow` for Tree paths and for Table columns,
with `offset`, `limit`, optional Table TAG filter, and optional batch size.
`TsFileTreeReader::queryByRow` accepts device and measurement lists.
- Python exposes `query_tree_by_row` and `query_table_by_row`; Table queries
accept `tag_filter` and `batch_size`.
- Treat `limit < 0` as unlimited and reject negative offsets. Close or destroy
every result set through the binding that created it.

## Timeseries Metadata

- C++ `TsFileReader::get_timeseries_metadata()` returns metadata for all
devices; its device-list overload restricts the request.
- Python `get_timeseries_metadata(device_ids=None)` returns a mapping keyed by
the full device-segment tuple. `None` selects all devices, while `[]` returns
an empty map; null TAG segments remain `None`.
- C callers must use the all-devices or selected-devices declarations and the
matching free routine in `tsfile_cwrapper.h`.

## Java TsBlock Write

Use `TableTsBlock2TsFileWriter` when the input is already a Table-model
`TsBlock`. Its constructor maps time, TAG, and FIELD column indexes and can
generate a monotonically increasing time column per device. Feed complete
blocks through `write(TsBlock)`, then close the writer. Build the mapping from
the current `TableSchema`; do not infer constructor positions from memory.

## Java Tablet Object and Size APIs

- Use `Tablet.addObjectPathValue` only for an `OBJECT` column; overloads accept
a measurement name or column index and a `String` or `byte[]` path.
- Use `Tablet.serializedSize()` to allocate or validate the serialized Tablet
payload. Do not replace it with a RAM-size estimate.

## Custom and Buffered Java Input

`TsFileSequenceReader` accepts a `TsFileInput`, enabling custom storage/input
implementations. For local buffered reads, construct `BufferedTsFileInput(Path,
bufferSize)` and pass it to the reader. Buffer size must be positive, and the
reader owns and closes the supplied input if initialization fails or when the
reader is closed.

## C++ Recovery and Append

Use `storage::RestorableTsFileIOWriter` to inspect and optionally truncate an
incomplete file before continuing a write. After `open(path, true)`, require
`has_crashed()` and `can_write()` before constructing `TsFileTreeWriter` or
`TsFileTableWriter` with the restorable writer. The recovered schema is used;
do not register a replacement schema. A complete file reports
`can_write() == false`. Keep the restorable writer alive longer than the facade
writer and preserve increasing timestamps beyond the recovered last values.

## Java BitMap Ranges

Use `markRange(start, length)` and `unmarkRange(start, length)` for mutations,
and `isRangeAnyMarked`, `isRangeAllMarked`, or `isRangeNoneMarked` for range
tests. Ranges are `(start, length)`, not `(start, end)`; empty ranges return
`false`, `true`, and `true`, respectively. Out-of-bounds ranges throw.

## Source Anchors

- Row queries: `cpp/src/reader/tsfile_reader.h`,
`cpp/src/reader/tsfile_tree_reader.h`, and `python/tsfile/tsfile_reader.pyx`
- Metadata: those readers, `cpp/src/cwrapper/tsfile_cwrapper.h`,
`python/tsfile/schema.py`, and `python/tests/test_reader_metadata.py`
- TsBlock: `java/tsfile/src/main/java/org/apache/tsfile/write/v4/TableTsBlock2TsFileWriter.java`
- Tablet: `java/tsfile/src/main/java/org/apache/tsfile/write/record/Tablet.java`
- Input: `TsFileSequenceReader.java`, `TsFileInput.java`, and
`BufferedTsFileInput.java` under
`java/tsfile/src/main/java/org/apache/tsfile/read/reader/`
- Recovery: `cpp/src/file/restorable_tsfile_io_writer.h` and the current C++
Tree/Table writer headers and recovery tests
- BitMap: `java/common/src/main/java/org/apache/tsfile/utils/BitMap.java` and
`java/tsfile/src/test/java/org/apache/tsfile/utils/BitMapTest.java`
83 changes: 83 additions & 0 deletions skills/tsfile/references/cpp-build-write.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# C++ Build and Parallel Write

## Shared or Static libtsfile

`TSFILE_BUILD_SHARED` selects the library kind and defaults to `ON`.

```sh
# Maven profile: shared default or static override
./mvnw clean verify -P with-cpp
./mvnw clean verify -P with-cpp -Dtsfile.build.shared=OFF

# Direct CMake static build
cmake -S cpp -B cpp/build/static \
-DTSFILE_BUILD_SHARED=OFF -DBUILD_TEST=OFF
cmake --build cpp/build/static --target tsfile

# Repository wrapper
cd cpp && bash build.sh --build-static
```

Use a separate build directory when switching library kind. A direct MSVC
consumer of the static archive must define `TSFILE_STATIC`; consumers of the
exported CMake target inherit the required definition. Verify the artifact in
`cpp/target/build/lib` for Maven or the selected CMake build tree.

`ENABLE_SIMD` also defaults to `ON` in the current CMake and Maven builds. Set
it explicitly only for portability diagnosis or controlled benchmarking; it
is independent of static/shared selection and thread support.

## Parallel Write

Compile thread support with `ENABLE_THREADS=ON` (the default). `OFF` strips the
threaded paths. Configure the current global worker pool through
`common/global.h`:

```cpp
#include "common/global.h"

common::set_parallel_write_enabled(true);
int status = common::set_thread_count(4); // valid range: 1..64
```

The current baseline uses `common::set_parallel_write_enabled` and
`common::set_thread_count`; use those exact names and namespace. The pool is
shared by parallel read and write paths. Configure it before creating active
readers/writers. Changing the count after initialization rebuilds the pool and
must never race with an operation using it.

Current source initializes parallel writing as enabled and the pool size as 6.
Do not substitute a hardware-concurrency default from older prose. Parallel
work is used only when the compiled path and runtime conditions qualify; keep
the tablet/batch write contract correct when it falls back to serial work.

Always check write return codes. If a multi-column batch fails partway, stop
using that writer rather than attempting to flush a potentially misaligned
partial batch.

## Source Anchors

- Link selection: `cpp/CMakeLists.txt`, `cpp/src/CMakeLists.txt`,
`cpp/build.sh`, `cpp/pom.xml`, and `cpp/README.md`
- Runtime API and defaults: `cpp/src/common/global.h`,
`cpp/src/common/global.cc`, and `cpp/src/common/config/config.h`
- Write behavior: `cpp/src/writer/tsfile_writer.cc` and current writer tests
Loading
Loading