K2GO-386 feat(app-backstop): firehose signal as a second reap trigger (L3a) - #551
Merged
Conversation
… server half)
The in-box guard (1.3.0) truncates a runaway .log every tick, so the disk may
never go low. But a recurring firehose is an off-proot orphan the box cannot
stop; only an app-side reap can. The app needs a SECOND reap trigger beyond its
own low-disk read.
Add GET /system/disk-guard/firehose returning
{ recurring, maxStreak, paths, lastTruncatedAtMs, now }. It reads the guard's
LIVE in-memory streak map, never a parsed log line, so a restart-resolved
firehose reports clean (ADR-386 §6, confirm before acting). recurring is
maxStreak >= 2. lastTruncatedAtMs lets the app judge freshness; the app
re-probes live growth before it reaps.
log-rotate.ts: track lastTruncatedAtMs; add pure summarizeStreaks (unit-tested)
and getFirehoseState wrapping it. Version 1.3.0 -> 1.3.1 (new REST endpoint).
… (L3a app half) The low-disk trigger catches a disk that already went low. But the in-box guard truncates a runaway .log every tick, so an off-proot orphan can firehose forever without the disk ever going low. The app needs to reap that orphan early. Add a second trigger to the guard poller. checkFirehoseSignal reads dash-node's live signal (GET /system/disk-guard/firehose). FirehoseSignal.isFresh gates on a recurring firehose whose last truncation is recent, judged in the server's own clock (no app-vs-server skew). The signal is only an ALERT: before reaping, the app re-probes live growth -- it sums the .log bytes under /var/log, waits, sums again, and acts only on a firehose-sized delta. So it never acts on a stale signal (confirm before acting, ADR-386 section 6). On a confirmed firehose it reaps + reclaims + relaunches (restart-to-keep-alive); the app-side reap reaches the off-proot orphan, so a fresh box does not refill. It does NOT touch the low-disk escalation counter -- that path stays the single escalation authority. Both paths skip the reap while a deep op holds the box. FirehoseSignal (pure, unit-tested) + FirehoseSignalSource (HttpURLConnection, returns null when the box is off). Debug receiver gains --ez firehose true to exercise the path on device without a real 20-min streak. Domain tests green (FirehoseSignal 5, Escalation 5, Policy 4). See ADR-386 section 6 + 10.
…; unify log walk (L3a review) Second code-review pass on L3a. Four fixes. 1. Growth threshold. GROWTH_MIN_BYTES 32 MiB -> 16 MiB. The observed firehose runs ~600 MB/min to 1.3 GB/min; at the low end that is ~30 MB in the 3s probe, which is under 32 MiB, so a slow-but-real firehose failed the confirm and was never reaped. 16 MiB (~327 MB/min) catches the low end with margin and stays far above any normal log. 2. Poll cadence. The firehose signal is an HTTP GET, and dash-node only advances it on its 10-min guard tick, so read it every 6th tick (~150s) instead of every 25s. The low-disk StatFs check still runs every tick. 3. Log walk. One recursive collectLogs(dir, out); totalLogBytes (sum) and biggestLog (max) reduce over it. Removes the duplicate walker (sumLogBytes vs biggestLogUnder). 4. Comment honesty. confirmFirehoseGrowing no longer claims to be robust to a truncation mid-probe: that rare race reads as no-growth this tick, not a false reap, and the next tick catches it.
…tics A disk-guard event is an operational diagnostic, not behavioural analytics. report() used DeliveryManager.enqueueAnalytics, which is gated on AnalyticsConsent (opt-in, default OFF) -- so by default the report was silently dropped, and the channel was wrong in kind. Send it to GlitchTip via Sentry instead (Sentry.withScope + captureMessage, WARNING), with the fields as tags/extras. Gated by CrashReportConsent (default ON, its own policy, no PII -- IIABApplication, ADFA-4533). A no-op when crash reporting is off or Sentry has no DSN. This also drops the flat JSON envelope in favour of structured tags. ADR-386: section 7 rewritten to the two-channel model (automatic -> GlitchTip; active user-sent -> feedback email); section 12 added to scope the active report as the closing piece of the effort (its own ticket). Verification table notes the full real firehose chain device-verified.
…/K2GO-386-firehose-signal
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A second reap trigger for the app-side backstop (ADR-386 section 6). dash-node
exposes a live firehose signal on
/system, and the app polls it to reap arecurring firehose the box cannot stop -- even before the disk goes low.
Stacked on
feat/K2GO-386-app-backstop(L3b). Review this after that one; thediff here is L3a only.
Why
L3b (the first trigger) reaps when free space already went CRITICAL. But Layer 2
truncates a runaway
.logevery tick, so the disk may never go low while anoff-proot orphan firehoses forever. The in-box guard can detect and reclaim, but
it cannot stop that orphan (device-proven). The app can. It needs to know a
firehose keeps coming back, so it can reap it early -- before the disk fills.
How
Server (dash-node):
GET /system/disk-guard/firehosereturns{ recurring, maxStreak, paths, lastTruncatedAtMs, now }. It reads the guard'sLIVE in-memory streak, never a parsed log line, so a restart-resolved firehose
reports clean.
recurringismaxStreak >= 2.lastTruncatedAtMsandnow(both server wall-clock) let the app judge freshness with no clock skew.
App:
FirehoseSignal(pure): the parsed signal plusisFresh, judged in theserver's clock. Unit-tested on a plain JVM.
FirehoseSignalSource: reads the endpoint; returns null when the box is off.DiskGuard.checkFirehoseSignal: the signal is only an ALERT. Before it reaps,the app RE-PROBES live growth -- it sums the
.logbytes under/var/log,waits, sums again, and acts only on a firehose-sized delta. So it never acts on
a stale signal (confirm before acting). On a confirmed firehose it reaps,
reclaims, and relaunches (restart-to-keep-alive). It does NOT touch the low-disk
escalation counter -- that path stays the single escalation authority.
dash-node's 10-min tick; the low-disk check still runs every tick.
Verification
FirehoseSignalTest(green); dash-nodelog-rotate.test.ts+tsc --noEmit(green).continuous writer made the guard truncate a >1 GiB log on two consecutive ticks
-> the endpoint went
recurring:true, maxStreak:2-> the app's poll read it,confirmed live growth, and reaped ("contained recurring firehose (streak 2)").
The box relaunched; the report carried
action:contained_firehose, trip:2(the real server streak, not the debug value). After the reap the endpoint reset
to
recurring:false-- the in-memory streak clears on restart, as designed.Coordination
feat/K2GO-386-app-backstop) and on Layer 2 (log rotation,merged in K2GO-386 feat(dashboard): proot-owned log rotation + firehose guard, dash-node-triggered #547). Retarget this PR to
mainonce L3b merges.(a null signal is a no-op) against an older dash-node.