Repository navigation
pkg/profile: do not write a function block the sizer did not budget - #6418
Open
guettlibot wants to merge 1 commit into
Open
guettlibot wants to merge 1 commit into
guettlibot wants to merge 1 commit into
Conversation
EncodeArrowLocation allocates its buffer from serializedArrowLocationSize and
then writes into it, so the two have to agree field for field. They did not.
The sizer budgets the per-line function block only when the line's function
name is valid:
size++ // 1 byte for whether there is a function
if lineFunctionName.IsValid(i) {
... start line, name, system name, filename ...
}
The writer emitted the flag as 0x1 and the block unconditionally. For a line
whose function name is null the sizer allowed 10 bytes, the writer needed more,
and it panicked in writeInt64AsUvarint writing the start line past the end of
the buffer.
A line with a number but no function is a representable shape, not a
corruption: DecodeInto emits exactly that from a pprof Line with FunctionId 0,
appending nulls to the function name, system name, filename and start line. Any
record carrying one through this encoder died, and EncodeArrowLocation is on
the normalizer's ingest path, which has no recover() -- so this was a failed
process rather than a failed request.
Writing 0x1 there was also a lie about the data even when the buffer happened
to be large enough: it claimed a function for a line that has none, with the
name read from whatever a null dictionary index resolves to.
The writer now mirrors the sizer's condition and writes 0x0 with no block when
there is no function name, which the decoders already round trip.
The test asserts the invariant rather than the symptom: for a line with a
function, a line without one, and both together, the bytes written must equal
the bytes budgeted and must decode cleanly through DecodeInto. Length alone
would not catch a writer that wrote too few bytes, since the buffer is
allocated at the budget and would simply carry trailing zeroes -- decoding is
what shows the bytes are well formed.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Independent of #6415, #6416 and #6417 — branches from
main, touches onlypkg/profile/encode.go.What
EncodeArrowLocationallocates its buffer fromserializedArrowLocationSizeand then writes into it, so the two have to agree field for field. They don't.The sizer budgets the per-line function block conditionally (
pkg/profile/encode.go:423):The writer emits the flag and the block unconditionally (
:362):Reproduced
For a line whose function name is null, the sizer allows 10 bytes and the writer runs off the end:
Why that shape occurs
A line with a number but no function is representable, not a corruption —
DecodeIntoemits exactly that from a pprofLinewithFunctionId == 0(pkg/profile/decode.go:190-193):EncodeArrowLocationis on the normalizer's ingest path (pkg/normalizer/arrow.go:601), andgrep -rn 'recover()' pkg cmdfinds nothing there — so any record carrying such a line took the process down rather than failing a request.Writing
0x1was also a lie about the data even when the buffer happened to be big enough: it claims a function for a line that has none, with the name read from whatever a null dictionary index resolves to.Fix
The writer now mirrors the sizer's condition, writing
0x0and no block when there is no function name. The decoders already round trip the0x0case.Test
It asserts the invariant rather than the symptom: for a line with a function, a line without one, and both together, the bytes written must equal the bytes budgeted and must decode cleanly through
DecodeInto. Length alone wouldn't catch a writer that wrote too few bytes — the buffer is allocated at the budget and would simply carry trailing zeroes — so decoding is what shows the bytes are well formed.Verified it fails on
main(should not panic / index out of range [0] with length 0) and passes with the fix.go test(incl.-race),go vet,gofmt, license check all clean acrosspkg/profile,pkg/normalizer,pkg/clickhouse,pkg/parcacol,pkg/parca.