Skip to content

fix(go): take a heap snapshot, not a 30-second delta - #130

Merged
blue4209211 merged 1 commit into
mainfrom
fix/go-heap-profile-is-a-snapshot
Sep 19, 2026
Merged

blue4209211 merged 1 commit into
mainfrom
fix/go-heap-profile-is-a-snapshot

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Summary

Go heap profiles come back empty — the Profiler tab and any event evidence show a flamegraph with nothing in it:

Type: inuse_space
Duration: 30.02s, Total samples = 0
Showing nodes accounting for 0, 0% of 0 total

The request was /debug/pprof/heap?seconds=30. On the heap endpoint seconds does not mean "sample for this long" — Go samples, waits, samples again, and returns the difference. A service whose heap is steady across that window deltas to exactly zero.

A heap profile is a snapshot of what the process is holding, so it takes no duration. Dropping seconds and adding gc=1 — which collects first, so the snapshot describes live memory rather than whatever the last GC cycle left behind — turns the same request into a real profile.

Type of change

  • Bug fix

Test plan

Reproduced against a real Go service with a steady ~64MB live heap and net/http/pprof, then confirmed the fix on the same process:

$ curl '/debug/pprof/heap?seconds=30'   # what we sent before
236 bytes
Duration: 30s, Total samples = 0
Showing nodes accounting for 0, 0% of 0 total

$ curl '/debug/pprof/heap?gc=1'         # what we send now
2247 bytes
Showing nodes accounting for 73.81MB, 100% of 73.81MB total
   71.81MB 97.28%  main.main
    1.50MB  2.04%  runtime.mallocgc
    0.50MB  0.68%  net/textproto.initCommonHeader

The empty output is character-for-character what the dev cluster produced: profiles of llm-server alternated between ~300 and ~16000 base64 bytes across four runs, the small ones decoding to a valid pprof with only sample-type headers and zero samples — i.e. whenever the heap happened to be steady for the 30s window.

go build ./... and go vet pass. Not unit-testable without an HTTP fixture: the URL is built inline in heapProfile and the function shells out to nsenter/wget.

Checklist

  • go build ./... passes locally
  • No breaking changes to the agent's stdout JSON envelope

Related issues

The last of the profiler defects from this series: #128 (image shipped no async-profiler), #129 (read-only rootfs / non-root targets), nudgebee/k8s-agent#619 (language detection).

Every Go heap profile the collector produced could come back empty:

    Type: inuse_space
    Duration: 30.02s, Total samples = 0
    Showing nodes accounting for 0, 0% of 0 total

The URL was /debug/pprof/heap?seconds=30. On the heap endpoint the seconds
parameter does not mean "sample for this long" - Go samples, waits, samples
again and returns the DIFFERENCE. A service whose heap is steady across the
window deltas to exactly zero, so the profile renders as an empty
flamegraph and the pod's Profiler tab shows nothing.

A heap profile is a snapshot of what the process is holding, so it takes no
duration. Dropping seconds and adding gc=1 - which collects first, so the
snapshot describes live memory rather than whatever the last GC left -
turns the same request into a real profile.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the heap profiling URL in internal/agent/profiler/go_pprof.go to request a snapshot of live memory using ?gc=1 instead of a delta over a duration. Feedback suggests correcting the added comment to accurately describe Go's heap profiling behavior regarding garbage collection and to warn about potential latency spikes caused by forcing a garbage collection on large heaps.

Comment thread internal/agent/profiler/go_pprof.go
@blue4209211
blue4209211 merged commit 2ca753a into main Sep 19, 2026
11 checks passed
@blue4209211
blue4209211 deleted the fix/go-heap-profile-is-a-snapshot branch September 19, 2026 04:02
mayankpande88 added a commit to nudgebee/k8s-agent that referenced this pull request Sep 19, 2026
ca8a3b3 asks Go for /debug/pprof/heap?seconds=30, which returns the delta
over that window rather than a snapshot - so every heap profile of a
service with a steady heap came back with Total samples = 0 and rendered
as an empty flamegraph. 2ca753a is the first tag with
nudgebee/application-profiler#130, which takes the snapshot instead.

No version bump: 0.1.25 is the latest release and 0.1.26 is still
unreleased, so this correction belongs in 0.1.26 rather than burning
another number for a chart nobody has installed yet.
mayankpande88 added a commit to nudgebee/k8s-agent that referenced this pull request Sep 19, 2026
…es (#623)

* fix(chart): pick up the profiler image whose heap profiles have samples

ca8a3b3 asks Go for /debug/pprof/heap?seconds=30, which returns the delta
over that window rather than a snapshot - so every heap profile of a
service with a steady heap came back with Total samples = 0 and rendered
as an empty flamegraph. 2ca753a is the first tag with
nudgebee/application-profiler#130, which takes the snapshot instead.

No version bump: 0.1.25 is the latest release and 0.1.26 is still
unreleased, so this correction belongs in 0.1.26 rather than burning
another number for a chart nobody has installed yet.

* chore: update image tags for main release

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Shiv <3078106+blue4209211@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants