Skip to content

pkg/profile: do not write a function block the sizer did not budget - #6418

Open
guettlibot wants to merge 1 commit into
parca-dev:mainfrom
guettli:upstream-pr-arrow-encoder-overrun
Open

guettlibot wants to merge 1 commit into
parca-dev:mainfrom
guettli:upstream-pr-arrow-encoder-overrun

Conversation

@guettlibot

Copy link
Copy Markdown

Independent of #6415, #6416 and #6417 — branches from main, touches only pkg/profile/encode.go.

What

EncodeArrowLocation allocates its buffer from serializedArrowLocationSize and then writes into it, so the two have to agree field for field. They don't.

The sizer budgets the per-line function block conditionally (pkg/profile/encode.go:423):

size++ // 1 byte for whether there is a function
if lineFunctionName.IsValid(i) {
	size = addSerializedInt64AsUvarintSize(size, lineFunctionStartLine.Value(i))
	size = addSerializedStringSize(size, ...name...)
	size = addSerializedStringSize(size, ...systemName...)
	...filename...
}

The writer emits the flag and the block unconditionally (:362):

buf[offset] = 0x1
offset++

offset = writeInt64AsUvarint(buf, offset, lineFunctionStartLine.Value(i))
offset = writeString(buf, offset, ...name...)
...

Reproduced

For a line whose function name is null, the sizer allows 10 bytes and the writer runs off the end:

sizer budget for the NULL-function line: 10 bytes

panic: runtime error: index out of range [0] with length 0
github.com/parca-dev/parca/pkg/profile.writeInt64AsUvarint(...)
	pkg/profile/encode.go:458
github.com/parca-dev/parca/pkg/profile.EncodeArrowLocation(...)
	pkg/profile/encode.go:365

Why that shape occurs

A line with a number but no function is representable, not a corruption — DecodeInto emits exactly that from a pprof Line with FunctionId == 0 (pkg/profile/decode.go:190-193):

} else {
	lw.FunctionStartLine.AppendNull()
	lw.FunctionName.AppendNull()
	lw.FunctionSystemName.AppendNull()
	lw.FunctionFilename.AppendNull()
}

EncodeArrowLocation is on the normalizer's ingest path (pkg/normalizer/arrow.go:601), and grep -rn 'recover()' pkg cmd finds nothing there — so any record carrying such a line took the process down rather than failing a request.

Writing 0x1 was also a lie about the data even when the buffer happened to be big enough: it claims a function for a line that has none, with the name read from whatever a null dictionary index resolves to.

Fix

The writer now mirrors the sizer's condition, writing 0x0 and no block when there is no function name. The decoders already round trip the 0x0 case.

Test

It asserts the invariant rather than the symptom: for a line with a function, a line without one, and both together, the bytes written must equal the bytes budgeted and must decode cleanly through DecodeInto. Length alone wouldn't catch a writer that wrote too few bytes — the buffer is allocated at the budget and would simply carry trailing zeroes — so decoding is what shows the bytes are well formed.

Verified it fails on main (should not panic / index out of range [0] with length 0) and passes with the fix. go test (incl. -race), go vet, gofmt, license check all clean across pkg/profile, pkg/normalizer, pkg/clickhouse, pkg/parcacol, pkg/parca.

EncodeArrowLocation allocates its buffer from serializedArrowLocationSize and
then writes into it, so the two have to agree field for field. They did not.
The sizer budgets the per-line function block only when the line's function
name is valid:

	size++ // 1 byte for whether there is a function
	if lineFunctionName.IsValid(i) {
		... start line, name, system name, filename ...
	}

The writer emitted the flag as 0x1 and the block unconditionally. For a line
whose function name is null the sizer allowed 10 bytes, the writer needed more,
and it panicked in writeInt64AsUvarint writing the start line past the end of
the buffer.

A line with a number but no function is a representable shape, not a
corruption: DecodeInto emits exactly that from a pprof Line with FunctionId 0,
appending nulls to the function name, system name, filename and start line. Any
record carrying one through this encoder died, and EncodeArrowLocation is on
the normalizer's ingest path, which has no recover() -- so this was a failed
process rather than a failed request.

Writing 0x1 there was also a lie about the data even when the buffer happened
to be large enough: it claimed a function for a line that has none, with the
name read from whatever a null dictionary index resolves to.

The writer now mirrors the sizer's condition and writes 0x0 with no block when
there is no function name, which the decoders already round trip.

The test asserts the invariant rather than the symptom: for a line with a
function, a line without one, and both together, the bytes written must equal
the bytes budgeted and must decode cleanly through DecodeInto. Length alone
would not catch a writer that wrote too few bytes, since the buffer is
allocated at the budget and would simply carry trailing zeroes -- decoding is
what shows the bytes are well formed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant