Skip to content

pkg/clickhouse: read the column, stop dropping function names - #6415

Open
guettlibot wants to merge 1 commit into
parca-dev:mainfrom
guettli:upstream-pr-location-column
Open

guettlibot wants to merge 1 commit into
parca-dev:mainfrom
guettli:upstream-pr-location-column

Conversation

@guettlibot

Copy link
Copy Markdown

What

decodeLineInfo in the ClickHouse backend does not read the column that the location encoders write between the line number and the hasFunction flag, so it reads the column's first byte as that flag.

For pprof, which carries no column information, that byte is a uvarint zero — the flag reads false and the whole function block (name, system name, filename, start line) is skipped.

How it shows up

A Go service's goroutine profile comes back with every frame unsymbolized, while scraping the same /debug/pprof endpoint directly names all of them. Nothing errors: the address, the mapping and the line number all decode, the row is written, and the profile is simply nameless. Measured on a server running v0.29.1 with --clickhouse-enabled before this change:

3,632,819 stored frames
  addresses decoded:    3,632,819
  line numbers decoded: 3,632,819
  function names:       0

Because the affected profiles are the ones that arrive already symbolized, the symbolizer cannot recover them afterwards — they generally carry no build ID to look up — so it reads like a debuginfo coverage gap rather than a decoder that disagrees with its own encoder.

Non-zero columns are worse than lossy

OTLP and Arrow locations carry real column numbers. There the misread desynchronises the remainder of the record rather than just losing the flag:

column=1                  -> name decoded as "\tmain.main"   (silently corrupt)
column=1, startLine=1000  -> panic: slice bounds out of range [:1012] with capacity 64
column=300                -> 0xAC != 0x01, falls back to names-dropped

There is no recover() on the ClickHouse ingest path, so that panic takes the process down rather than failing a request.

Why this is the decoder's bug

profile.DecodeInto — the canonical decoder — reads line, then column, then the flag (pkg/profile/decode.go). Every encoder writes them in that order: EncodePprofLocation, EncodeOtelLocation, EncodeArrowLocation, and the normalizer's v2 encoder (pkg/normalizer/arrow_v2.go, whose format comment spells out uvarint(lineNumber), uvarint(column) then byte hasFunction). decodeLineInfo was the only parser in the tree that disagreed, which is also why the same one-line read fixes the pprof, OTLP and v2 paths together and changes none of them in any other way.

Tests

TestDecodeLineInfoRoundTripsTheEncodedLocation drives the real encoders rather than hand-built blobs — a blob written by hand would be written from whatever the decoder happens to do, and so would agree with the bug it exists to catch.

Coverage:

  • pprof, with and without a mapping (the mapping block sits between the line count and the lines, so a decoder wrong about one can easily be right about the other)
  • OTLP with non-zero columns 1, 7, 300, 16384 — a zero column is a single 0x00 byte and cannot tell a uvarint read apart from a bare offset++; the multi-byte values pin down that it is a uvarint read
  • a line with no function (FunctionId == 0), whose fields must come back zeroed rather than dropped

Against main the pprof cases fail with the name discarded and the OTLP case panics; with the fix all pass. go vet ./pkg/clickhouse/ is clean and ./pkg/clickhouse, ./pkg/profile, ./pkg/normalizer pass.

Note

Deliberately minimal. Two adjacent things are left alone and can follow separately if you'd like them:

  1. decodeLineInfo has no bounds checks at all, so a short or malformed record panics on an ingest path with no recovery. This change narrows the window but does not close it.
  2. The ClickHouse backend keeps only Line[0], so inlined frames above the innermost one are dropped at ingest — pre-existing, and orthogonal to the wire format.

A Go service's goroutine profile came back from a server running the
ClickHouse backend with every frame unsymbolized, while scraping the very
same /debug/pprof endpoint directly named all of them. The frame count
survived ingestion; the names did not.

The location encoders write a column between the line number and the
hasFunction flag. decodeLineInfo did not read it, so it read the column's
first byte as the flag. For pprof, which carries no column information, that
byte is a uvarint zero, so the flag read as false and the whole function
block -- name, system name, filename, start line -- was skipped.

Nothing failed while this happened. The address, the mapping and the line
number all decoded, the row was written, and the profile came back nameless.
Measured on a live server before this change: 3,632,819 stored frames, every
address and every line number decoded, zero function names.

A non-zero column was worse than lossy. OTLP and Arrow locations carry real
column numbers, and there the misread desynchronised the rest of the record:
column 1 decoded a corrupted name ("\tmain.main"), and a larger start line
walked the offset off the end of the buffer and panicked. The ClickHouse
ingest path has no recover(), so that is a dead server rather than a failed
request.

Only profiles that arrive already symbolized lose names -- anything scraped
from a Go /debug/pprof endpoint, and any other pre-symbolized upload. Those
are also the ones the symbolizer cannot rescue afterwards, because they
usually carry no build ID to look up, so the result reads like a debuginfo
coverage gap rather than a decoder that could not parse what it was handed.

profile.DecodeInto, the canonical decoder, reads line then column then the
flag; every encoder in pkg/profile writes them in that order. decodeLineInfo
was the only parser in the tree that disagreed.

The tests drive the real encoders rather than hand-built blobs, because a
blob written by hand would be written from whatever the decoder happened to
do and so would agree with the bug it existed to catch. They cover the
mapping and no-mapping branches, a line with no function, and non-zero
columns through EncodeOtelLocation -- including multi-byte ones, since a zero
column cannot tell a uvarint read apart from a bare offset++.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant