ClickHouse backend: v0.29.0 panics during flamegraph queries (arrow/array: index out of range) #6409

Description

Parca v0.29.0 crashes when executing a merge flamegraph query against its ClickHouse backend. The server starts successfully and passes health probes, but querying existing, symbolized profiles terminates the process.

This reproduces through both Parca’s HTTP API and Grafana’s Parca datasource. We are using the official image without source modifications.

Environment

• Parca: ghcr.io/parca-dev/parca:v0.29.0
• Commit: b2956713c1f56d7dca98b9f3582ea59e84b8ace7
• Platform: Kubernetes, Linux/amd64
• ClickHouse: 26.8.10.6
• Backend: ClickHouse enabled; no Parca scraping or ingestion

Profiles are ingested separately into an OpenTelemetry profile table. Parca reads an ordinary SQL compatibility view exposing its expected ClickHouse schema, including populated function names and corresponding stacktrace arrays. This failure was observed with that view; we have not independently reproduced it using Parca’s own ingestion path.

Steps to reproduce

1. Start the official v0.29.0 image with the ClickHouse backend configured:

   --clickhouse-enabled
   --clickhouse-address=<clickhouse-host>:9000
   --clickhouse-database=profiling
   --clickhouse-table=parca_otel_profiles
   --clickhouse-username=<username>
   

2. Ensure the configured table/view contains symbolized CPU samples.

3. Execute a merge query covering existing samples. Replace the timestamps and service selector as appropriate:

   curl --get 'http://localhost:7070/api/profiles/query' \
     --data-urlencode 'mode=MODE_MERGE' \
     --data-urlencode 'merge.query=process_cpu:cpu:nanoseconds:cpu:nanoseconds:delta{service_name="java-profiles-app"}' \
     --data-urlencode "merge.start=$START" \
     --data-urlencode "merge.end=$END" \
     --data-urlencode 'report_type=REPORT_TYPE_FLAMEGRAPH_ARROW'

Expected behavior

Parca returns the flamegraph response without terminating.

Actual behavior

The process panics and Kubernetes restarts the container. Repeated dashboard queries trigger repeated crashes.

Abbreviated stack trace:

panic: arrow/array: index out of range

github.com/apache/arrow-go/v18/arrow/array.NewSliceData(...)
    arrow/array/data.go:232
github.com/apache/arrow-go/v18/arrow/array.(*Struct).setData(...)
    arrow/array/struct.go:195
github.com/apache/arrow-go/v18/arrow/array.(*StructBuilder).NewStructArray(...)
    arrow/array/struct.go:406
...
github.com/apache/arrow-go/v18/arrow/array.(*RecordBuilder).NewRecordBatch(...)
    arrow/array/record.go:371
github.com/parca-dev/parca/pkg/clickhouse.(*Querier).rowsToArrowRecords(...)
    pkg/clickhouse/querier.go:1069
github.com/parca-dev/parca/pkg/clickhouse.(*Querier).QueryMerge(...)
    pkg/clickhouse/querier.go:661

AI Suspected cause

The Arrow profile writer includes a ColumnNumber builder. However, [rowsToArrowRecords](https://github.com/parca-dev/parca/blob/v0.29.0/pkg/clickhouse/querier.go#L1000) appends line numbers and function fields without appending ColumnNumber, in both the symbolized-location and stored-function branches.

QueryMerge()
  → execute SQL against ClickHouse
  → rowsToArrowRecords()
  → profile.NewWriter()
  → append returned samples and stack frames
  → RecordBuilder.NewRecordBatch()
  → subsequent flamegraph processing

Version comparison and workaround

• v0.29.0: runtime failure confirmed.
• v0.28.0, v0.27.1, v0.27.0: their ClickHouse querier and Arrow writer files are identical to v0.29.0. These versions were inspected, not separately runtime-tested.
• v0.26.0: works with the same ClickHouse view and data. All eight tested Java/Go CPU and memory combinations returned nonempty flamegraphs with totals matching direct ClickHouse queries.

v0.26.0 predates the ColumnNumber writer field and is our temporary workaround.

Could the ClickHouse-to-Arrow conversion ensure every line field is populated consistently, with a regression test covering symbolized samples and REPORT_TYPE_FLAMEGRAPH_ARROW queries?