compute: expose replica resource usage observations

[antiguru]

Motivation

Replica resource usage is only visible as periodic orchestrator samples, roughly one per minute, landing in mz_cluster_replica_metrics_history. A spike between two samples is invisible, so the usage of a hydration episode that starts and finishes inside one sampling gap cannot be recovered at all. This is the replica-local, higher-cadence half of the Improved Hydration Visibility work.

Description

mz_metrics::usage reads cgroup v2 interface files, getrusage, /proc/self/status and statvfs on the existing periodic metrics task. A new ComputeLog::ResourceUsage logging dataflow reports them as mz_introspection.mz_cluster_replica_resource_usage(process_id, source, metric, value), following the ComputeLog::PrometheusMetrics shape: usage is per-process, so one worker per process reports and the rest drop their capability.

Three decisions the diff cannot show.

The relation surfaces uninterpreted observations. It does not decide which number is "the" memory usage of a replica, and it never combines two sources into a third figure. Sources measure overlapping but distinct quantities: cgroup memory.current is the accounting that limit enforcement and the OOM killer act on, while VmRSS is the process's resident set. They disagree by a measured ~96 MiB on staging, in the direction of VmRSS being the larger, because a page is charged to whichever cgroup first faulted it in and this binary's resident text was faulted in by the runtime that unpacked the image. That disagreement is the diagnostic signal, and it is not recoverable once two sources have been folded into one number. Combining sources belongs in views, where it can change without a migration and where a reader can see what was combined. For what fusing costs, note that mz_cluster_replica_metrics.disk_bytes currently reports disk plus swap because a consumer wanted it displayed that way, and the fix is blocked on porting that consumer.

Long rows, not one column per metric. A wide row retracts and re-inserts on any field's change, so its churn is set by its noisiest field: a filesystem-usage number that moves every sample would drag a memory peak that has not moved in an hour through a retraction every time. Long rows churn only what changed. They also make a missing observation representable as an absent row rather than an in-band sentinel, and let a new metric be added as kernels gain interface files without a catalog migration.

Peaks are reported, not derived. A peak cannot be recovered downstream from this relation: each sample retracts the previous one, so a reduce computing a maximum sees one value per key and returns it. Retaining the samples instead is not an option, because a compute log collection is an arrangement with no retention, window or TTL mechanism available to it, so a retained history would grow unbounded for the process's lifetime. Most peaks cost nothing to report because the kernel maintains them: cgroup memory.peak and memory.swap.peak are exact high-water marks of the cgroup accounting, kept per cgroup for the container's lifetime and therefore unaffected by how rarely we read them, by the sampler being disabled and restarted, or by environmentd restarting; ru_maxrss is already a high-water mark. Only a source with no kernel-side peak gets one folded in the sampler, currently the scratch filesystem and swap below the kernel version providing memory.swap.peak, and those are reported under a distinct _peak metric name so a reader can tell an exact peak from a sampled one.

The logging operator downgrades its capability on its own timer rather than on the sampler's. Were the sampler to drive progress, disabling it (which mz_metrics_usage_refresh_interval = 0 is meant to allow) would freeze the collection's frontier and wedge every query over the relation along with the mz_catalog_server indexes on it.

Cadence is a new dyncfg, mz_metrics_usage_refresh_interval (5s), kept separate from memory_limiter_interval, which governs OOM-kill behavior and must not be retuned for introspection's sake.

Cost

No source read here is proportional to heap size. VmRSS and VmSwap come from per-mm counters and the cgroup figures are page_counter reads. Measured on a 32-core Linux host, p50 over 2000 iterations:

read	empty heap	32 GiB resident
getrusage(RUSAGE_SELF)	0.22 us	0.22 us
statvfs	0.41 us	0.41 us
/proc/self/status	4.67 us	4.74 us
/proc/self/smaps_rollup	9.6 us	137 ms
/proc/self/numa_maps	20.8 us	142 ms

The last two rows are not read by this path and must not be added to it. They walk page tables, so their cost scales with resident pages and rises superlinearly. Held at 16 GiB resident, numa_maps costs 28.9 ms over one fully-touched VMA, 11.0 ms over 8192 VMAs, and 1.5 ms when only every 64th page is written, confirming the cost tracks page-table entries rather than VMA count.

At the 5s default the whole sampler is roughly 0.0002% of one core.

Incidental refactors

ProcStatus and the statvfs helper move from mz_compute::memory_limiter to mz_metrics::usage, so the limiter, the clusterd usage collector and the sampler share one reader. The cgroup helpers move from mz_ore::memory to mz_ore::cgroup and lose their target_os = "linux" gate, since a second crate now uses them and the file is pure std. The module gains a CgroupV2 handle that resolves the cgroup directory once and reads its interface files, rather than this becoming a fourth copy of the /proc/self/cgroup parse. It exposes the resolved path rather than logging it, so the module keeps its property of depending on nothing outside std: it is compiled unconditionally now, including into the wasm32 build where mz_ore's optional dependencies are absent. The three existing cgroup limit readers (usage_metrics.rs, ore::cgroup, service::boot) are deliberately left alone; consolidating them is worth doing separately.

The Prometheus metrics operator reports a sampled source in the same shape, so the interval-boundary capability downgrade and the snapshot diff become downgrade_to_interval_boundary and emit_snapshot_diff in mz_compute::logging, and both operators call them.

Docs: the relation is described in mz_introspection.md, and the sources, the metrics they report, how to interpret them and an example query live on a new page, manage/monitor/replica-resource-usage.md.

Adding a builtin log changes the inlined-VALUES fingerprints of mz_catalog.mz_indexes and mz_catalog.mz_sources, so both get a MigrationStep::replacement at the current dev version.

Verification

• Unit tests in mz_metrics::usage: a sample is non-empty, a derived peak rises with a higher observation and survives a lower one, an unreadable source is absent rather than zero, and a derived peak is only published for a source that was actually read.
• test-resource-usage in test/cluster/mzcompose.py: asserts both processes of a 2-process replica report, that they report the same metric set, that rusage and proc_status are present, and that no peak goes backwards across a workload. cgroup metrics are deliberately not asserted, since memory.peak and memory.swap.peak depend on the kernel version.
• Catalog goldens regenerated. Note that autogenerated/mz_introspection.slt needed ci/test/lint-docs-catalog.sh --rewrite, not just --rewrite-results: the relation name is baked into the query text, and a plain rewrite silently produced an empty result.
• Verified on staging (eu-west-1). memory_peak and swap_peak are both present on the node kernel, and cgroup memory_current matches cAdvisor's container_memory_working_set_bytes for the same container to within 1.5% across every pod, so the cgroup directory resolution is correct.
• A replica hydrating a 60M-row index against a 3.79 GiB RAM limit reported memory_current 1.20 GiB / swap_current 695 MiB while memory_peak was 3.79 GiB and swap_peak 3.98 GiB: the instantaneous readings understate the episode by 3.2x and 5.9x. Sampled again minutes later, memory_current had risen 2.46 GiB and swap_current fallen 616 MiB as pages swapped back in, with both peaks byte-identical. That is the monotonicity property under a bidirectional swing.
• VmRSS runs a roughly constant 96 MiB above the replica's own cgroup charge (7% spread across replicas spanning 36 MiB to 400 MiB of memory_current), because this binary's resident text is charged to the runtime that unpacked the image. Taking ru_maxrss as the replica's footprint, as the previous design did, overstates the enforced figure by 2.75x on an idle replica and 1.20x on a loaded one.
• memory.events' max stayed 0 even with the cgroup pinned at its ceiling, because reclaim succeeded by swapping rather than failing. It is therefore not a reliable limit-hit signal where swap is configured; memory_peak reaching memory_max is. Both are reported without interpretation.

Release notes

This release adds mz_introspection.mz_cluster_replica_resource_usage, reporting the resource usage of each cluster replica process as measured by each available source, including cgroup memory and swap usage and their kernel-maintained peaks.

[linear-code]

CPU-211

[aljoscha]

Some of the text here gets a bit lost in the weeds. Plus, could we have a table that lists the sources and what metric they expose, so that I can have an overview of what's there?

[antiguru]

Restructured. The section now leads with two short paragraphs, then a table of the four sources and what each one measures, then a table of every metric with its source and meaning. The nuance that was buried in prose moved into the meaning column, and what is left as prose is a short "Interpreting" list of the questions people actually ask and which metric answers them.

I also added an example query, since a couple of people have asked for the same thing: how close each process came to a memory-limiter kill, comparing proc_status heap and heap_peak against mz_cluster_replica_metrics.heap_limit. Verified against staging.

_{Posted by Claude Code}

[aljoscha]

To me, most of the explanation is a bit to dense to read with this measly human brain. I think the design that matters is mostly:

• we're adding a builtin collection, it lives on the replica
• it reports different metrics from different sources, so would be good to have a table of the sources and the metrics they report, maybe a matrix/2d table even?
• and we say that we have useful peak metrics because they prevent problems with sampling

and a lot of the other discussion can go into a technical appendix or be removed to a degree

But that depends on how useful we want the design doc to be for humans later on.

[antiguru]

Resolved by dropping the design doc entirely: the change is not contentious enough to need one. What you asked for as the readable core is now split by audience. The three decisions (builtin collection living on the replica, multiple sources, peaks reported rather than sampled downstream) are in the PR description, and the source and metric tables plus the interpretation guidance are a user-facing page, manage/monitor/replica-resource-usage.md.

_{Posted by Claude Code}

[DAlperin]

LGTM. Agree with Aljoscha on the design doc notes though if we can clean that up first

[antiguru]

A better structure could be to extract the detailed description and interpretation to a new page, which we link here. What stays is the description of the relation itself.

[antiguru]

Done. mz_introspection.md now carries only what the relation is: two short paragraphs plus the column table. The source table, the metric table, the interpretation list and the example query moved to a new page, manage/monitor/replica-resource-usage.md, linked from the section.

_{Posted by Claude Code}

[antiguru]

If documented, link the feature flag that controls the sampling frequency.

[antiguru]

It is not documented, and I did not add it. There is no user-facing page for dyncfgs at all, and mz_metrics_usage_refresh_interval is replica-scoped and not settable by a user, so naming it here would point readers at a knob they cannot reach. Instead the cross-reference now states the cadence: "sampled every few seconds" in both mz_cluster_replica_metrics and _history, against "roughly once a minute" for the orchestrator samples, which is the part a reader is actually choosing between.

_{Posted by Claude Code}

[antiguru]

I think we can do without a design doc. This change isn't contentious enough to need one.

[antiguru]

Removed. The parts worth keeping are already elsewhere: the three design decisions and the syscall cost table are in the PR description, and the reader-facing half is the new manage/monitor/replica-resource-usage.md page.

_{Posted by Claude Code}

[antiguru]

I think this is the same logic we have for the prometheus collector? If yes, extract a common function.

[antiguru]

Yes, twice over. Both were extracted into mz_compute::logging:

• downgrade_to_interval_boundary(cap, activator, now, start_offset, interval_ms) -> Timestamp for the capability downgrade plus the boundary-aligned reactivation.
• emit_snapshot_diff(session, packer, prev, current, ts, pack) for the retract-changed-and-removed, insert-changed-and-new loop, generic over the key and value types.

pack takes the packer as an argument instead of closing over it, because it returns rows borrowed from it and a closure cannot lend from its own captures.

_{Posted by Claude Code}

[antiguru]

Renamed the relation from mz_cluster_resource_usage to mz_cluster_replica_resource_usage.

The old name was wrong on the facts. The data is one replica's processes, the row carries process_id and no cluster or replica identifier, and the whole mz_introspection schema is replica-local, which is why every other relation there carries no scope prefix at all. Only mz_cluster_prometheus_metrics does, and this relation had copied it. cluster is also the wrong aggregation level for the questions this relation exists to answer: replicas of a cluster are independent copies, each with its own limit, so reading the number as cluster-wide gives a wrong conclusion rather than just an awkward name.

mz_cluster_replica_resource_usage uses the SQL surface's own object name (CREATE CLUSTER REPLICA, mz_cluster_replicas), joins the existing 11-relation mz_cluster_replica_* family, and makes the pairing with mz_internal.mz_cluster_replica_metrics explicit: same subject, one at roughly a minute across all replicas, the other every few seconds on one replica with peaks.

One wrinkle worth stating: every existing mz_cluster_replica_* relation carries a replica_id, and this one cannot, because it is replica-local introspection. The docs say to pin the replica.

OID stays 17123. ComputeLog::ResourceUsage and the operator name are unchanged, matching how ComputeLog::PrometheusMetrics already differs from its relation name. mz_cluster_prometheus_metrics shipped in March and is not renameable without a migration, so the two names diverge; one shipped mistake did not seem like a reason to add a second.

_{Posted by Claude Code}