Skip to content

benchmark_perf: report PSS alongside summed RSS - #55

Open
Rian354 wants to merge 1 commit into
mainfrom
rian/pss-benchmark-memory
Open

benchmark_perf: report PSS alongside summed RSS#55
Rian354 wants to merge 1 commit into
mainfrom
rian/pss-benchmark-memory

Conversation

@Rian354

@Rian354 Rian354 commented Sep 5, 2026

Copy link
Copy Markdown

Every harness in examples/benchmark_perf/ measured peak memory as parent RSS plus the summed RSS of all children:

total += self._proc.memory_info().rss
for child in self._proc.children(recursive=True):
    total += child.memory_info().rss

That double-counts copy-on-write pages shared between forked workers, and the error is not small. Measuring a 64-worker pandarallel ingest this way produced 1.89 TB on a 480 GB node — physically impossible, which is how the problem surfaced. The same accounting inflated our own PyHealth 2.0 worker sweep:

workers summed RSS PSS parent-only RSS
8 13,368 MB 12,283 MB 2,205 MB
16 22,661 MB 19,530 MB 2,122 MB
32 39,109 MB 32,861 MB 2,139 MB

PSS (proportional set size) divides each shared page among the processes mapping it, so the total is additive. It is the figure to quote when the question is "how much memory does this need."

What changed

Adds _total_pss_bytes() and a peak_pss_bytes field to all 16 harnesses — legacy, workers_n, meds_reader, and patient_exploration variants. Falls back to RSS per-process where PSS is unavailable (non-Linux, or insufficient permissions).

Additive on purpose. peak_rss_bytes keeps its existing meaning so previously-collected numbers remain interpretable; PSS lands beside it rather than silently replacing it. Summed RSS is also worth keeping as a negative control — the gap between the two columns is what demonstrates the sharing.

Not addressed here: torch.cuda.max_memory_allocated similarly understates GPU requirements (a cell reporting 5.9 GB allocated held 10.19 GiB and OOMed on a 10.57 GiB card), but that is a separate change in trainer.py.

Every harness peaked on parent RSS plus the summed RSS of all children. That
double-counts copy-on-write pages shared between forked workers, and the error
is not small: a 64-worker pandarallel ingest measured this way summed to
1.89 TB on a 480 GB node. The same accounting inflated our own PyHealth 2.0
worker sweep, where PSS shows 32.1 GB at 32 workers against 39.1 GB for
summed RSS.

Adds PSS (proportional set size -- each shared page divided among the
processes mapping it) across the process tree, with an RSS fallback where PSS
is unavailable. Additive: peak_rss_bytes keeps its existing meaning so numbers
already collected stay interpretable, and peak_pss_bytes lands beside it.

Applies to all 16 harnesses: legacy, workers_n, meds_reader, and
patient_exploration variants.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants