Repository navigation
pkg/clickhouse: read the column, stop dropping function names - #6415
Open
guettlibot wants to merge 1 commit into
Open
guettlibot wants to merge 1 commit into
guettlibot wants to merge 1 commit into
Conversation
A Go service's goroutine profile came back from a server running the
ClickHouse backend with every frame unsymbolized, while scraping the very
same /debug/pprof endpoint directly named all of them. The frame count
survived ingestion; the names did not.
The location encoders write a column between the line number and the
hasFunction flag. decodeLineInfo did not read it, so it read the column's
first byte as the flag. For pprof, which carries no column information, that
byte is a uvarint zero, so the flag read as false and the whole function
block -- name, system name, filename, start line -- was skipped.
Nothing failed while this happened. The address, the mapping and the line
number all decoded, the row was written, and the profile came back nameless.
Measured on a live server before this change: 3,632,819 stored frames, every
address and every line number decoded, zero function names.
A non-zero column was worse than lossy. OTLP and Arrow locations carry real
column numbers, and there the misread desynchronised the rest of the record:
column 1 decoded a corrupted name ("\tmain.main"), and a larger start line
walked the offset off the end of the buffer and panicked. The ClickHouse
ingest path has no recover(), so that is a dead server rather than a failed
request.
Only profiles that arrive already symbolized lose names -- anything scraped
from a Go /debug/pprof endpoint, and any other pre-symbolized upload. Those
are also the ones the symbolizer cannot rescue afterwards, because they
usually carry no build ID to look up, so the result reads like a debuginfo
coverage gap rather than a decoder that could not parse what it was handed.
profile.DecodeInto, the canonical decoder, reads line then column then the
flag; every encoder in pkg/profile writes them in that order. decodeLineInfo
was the only parser in the tree that disagreed.
The tests drive the real encoders rather than hand-built blobs, because a
blob written by hand would be written from whatever the decoder happened to
do and so would agree with the bug it existed to catch. They cover the
mapping and no-mapping branches, a line with no function, and non-zero
columns through EncodeOtelLocation -- including multi-byte ones, since a zero
column cannot tell a uvarint read apart from a bare offset++.
This was referenced Oct 2, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
decodeLineInfoin the ClickHouse backend does not read the column that the location encoders write between the line number and thehasFunctionflag, so it reads the column's first byte as that flag.For pprof, which carries no column information, that byte is a uvarint zero — the flag reads false and the whole function block (name, system name, filename, start line) is skipped.
How it shows up
A Go service's goroutine profile comes back with every frame unsymbolized, while scraping the same
/debug/pprofendpoint directly names all of them. Nothing errors: the address, the mapping and the line number all decode, the row is written, and the profile is simply nameless. Measured on a server running v0.29.1 with--clickhouse-enabledbefore this change:Because the affected profiles are the ones that arrive already symbolized, the symbolizer cannot recover them afterwards — they generally carry no build ID to look up — so it reads like a debuginfo coverage gap rather than a decoder that disagrees with its own encoder.
Non-zero columns are worse than lossy
OTLP and Arrow locations carry real column numbers. There the misread desynchronises the remainder of the record rather than just losing the flag:
There is no
recover()on the ClickHouse ingest path, so that panic takes the process down rather than failing a request.Why this is the decoder's bug
profile.DecodeInto— the canonical decoder — reads line, then column, then the flag (pkg/profile/decode.go). Every encoder writes them in that order:EncodePprofLocation,EncodeOtelLocation,EncodeArrowLocation, and the normalizer's v2 encoder (pkg/normalizer/arrow_v2.go, whose format comment spells outuvarint(lineNumber), uvarint(column)thenbyte hasFunction).decodeLineInfowas the only parser in the tree that disagreed, which is also why the same one-line read fixes the pprof, OTLP and v2 paths together and changes none of them in any other way.Tests
TestDecodeLineInfoRoundTripsTheEncodedLocationdrives the real encoders rather than hand-built blobs — a blob written by hand would be written from whatever the decoder happens to do, and so would agree with the bug it exists to catch.Coverage:
1, 7, 300, 16384— a zero column is a single0x00byte and cannot tell a uvarint read apart from a bareoffset++; the multi-byte values pin down that it is a uvarint readFunctionId == 0), whose fields must come back zeroed rather than droppedAgainst
mainthe pprof cases fail with the name discarded and the OTLP case panics; with the fix all pass.go vet ./pkg/clickhouse/is clean and./pkg/clickhouse,./pkg/profile,./pkg/normalizerpass.Note
Deliberately minimal. Two adjacent things are left alone and can follow separately if you'd like them:
decodeLineInfohas no bounds checks at all, so a short or malformed record panics on an ingest path with no recovery. This change narrows the window but does not close it.Line[0], so inlined frames above the innermost one are dropped at ingest — pre-existing, and orthogonal to the wire format.