K2GO-386 feat(dashboard): proot-owned log rotation + firehose guard, dash-node-triggered - #547
Merged
Merged
Conversation
…node proot has no systemd/cron, so /etc/cron.daily/logrotate never runs — logrotate is installed but nothing triggers it, and a service log (php-fpm, dash-node) can grow until the device hits ENOSPC. Own our rotation instead of the RPi-oriented inherited config: - tools/setup-proot-logging.sh: idempotent configurator. Moves the inherited nginx/php-fpm snippets out of /etc/logrotate.d (we override them; a duplicate path fails logrotate), installs /etc/logrotate.d/k2go (copytruncate + size-based, so no daemon reopen signal is needed under proot and a file that grows is rotated regardless of schedule), covering php-fpm, nginx, calibre-web, dash-node; validates with logrotate -d. kiwix writes no log; kolibri self-rotates — both left alone. - dash-node (static/dashboard): startLogRotation() installs the config once at startup (light: mv + write + validate, no rotation) and triggers logrotate every 10 min. No rotation at boot (first pass at +10 min), keeping the fragile boot moment clear. - ADR-386 documents the full disk-fill containment strategy and this layer. Typecheck/tests run in the on-device dashboard build (tsc + smoke test).
…-node boot; bump 1.3.0-dev.1 Move the K2Go log-rotation config install out of dash-node startup into the three deploy paths — iiab-android install_iiaboa_dashboard (rootfs build → a clean R2 rootfs ships preconfigured), rebuild-dashboard.sh and dev-push-dashboard.sh (updates re-assert it). Reconfiguring on every boot would be needless churn; installing on build/update is more defensive. dash-node keeps only the 10-min logrotate trigger (no rotation at boot). Bump dash-node 1.2.12 -> 1.3.0-dev.1 (-> 1.3.0 at merge): not a REST-surface change, but the version bump is how a new dash-node behavior is delivered to devices (the update pill; no ansible role yet), so a behavior change that must reach the field gets a bump.
Double-pass code review on the L2 diff: - setup-proot-logging.sh: validate the effective config and ROLL BACK to the exact pre-run state on failure (snapshot k2go + track moved snippets), so a validation failure never leaves a broken config that breaks all rotation. Move php*-fpm by glob (was hardcoded php8.4-fpm) to match the config's /var/log/php*-fpm.log — one source for 'which php', survives a version bump. Resolve logrotate by absolute path. - log-rotate.ts: call /usr/sbin/logrotate by absolute path (dash-node's runtime PATH may omit /usr/sbin -> ENOENT every tick). - server.ts: gracefulShutdown clears the timer via stopLogRotation() (explicit lifecycle). Tracked (not fixed): a package update recreating a default snippet between deploys (Phase-2 pdsm re-assert), and copytruncate of dash-rebuild.log mid-rebuild (size cap bounds it).
Add a focused usage() + `case "${1:-}" in -h|--help)` early-exit to the four shell
scripts this branch touches (setup-proot-logging.sh, rebuild-dashboard.sh,
dev-push-dashboard.sh, iiab-android), so their usage can be consulted without opening
the file. Uses a hand-written usage block (not `grep '^#'`) so inline comments and the
config heredoc don't leak into the help. Additive: existing args (branch, clone_dir, -f,
set-hostname) fall through unchanged; --help works before iiab-android's root check.
…half)
A firehose (a runaway logging ~GB/min) must never reach logrotate: copytruncate
would COPY a multi-GB file (doubling disk, pegging CPU on a weak phone) and still
not stop the writer. So dash-node's tick now runs, IN ORDER on ONE timer:
1. firehose guard — truncate any /var/log[/nginx]/*.log past ~1 GiB IN PLACE
(reclaim, no copy/compress; a firehose log is garbage). Deterministic order
(guard->rotate), one clock, so there is no second timer to drift.
2. logrotate — rotate the remaining moderate logs.
This is the in-box half of L3 (ADR-386 §6): dash-node can DETECT + RECLAIM but
cannot stop an off-proot orphan (device-proven); a recurring firehose is flagged
(occurrence count) for the app-side reap, which is the other half, still to build.
Truncating each tick bounds the disk even for an unstoppable orphan (headroom >
rate*interval), buying time for the app-side stop.
…ehose streak leak Double-pass code-review fixes on the firehose guard, verified with the real toolchain (tsc strict --noEmit exit 0; node --test 4/4 passing): - Extract the streak logic into a pure exported updateStreaks(firehosing, prev) and add sockets/log-rotate.test.ts (first-time / recurrence / drop-on-absent / empty). The guard now rebuilds the streak map from ONLY the paths firehosing this pass, so a deleted or renamed log no longer leaves a stale entry. Verified NOT issues (checked, not assumed): the try/catch definite-assignment on 'entries' compiles clean under strict tsc; guarding on statSync().size (logical) is correct because logrotate's copytruncate copies by logical size.
…ock-driven signal Capture the design rule Luis raised: a clock/calendar-driven signal always leaves gaps, so the actor must file the edges. Because dash-node cannot push (pull-only, verified) and every layer here is clock-driven, a reported firehose can be a tick — or a whole app restart — stale by the time the app reads it (user feels heat, closes + reopens the app, the firehose is gone, but an old dash-node.log line still says "under a busy-loop"). ADR-386 §6 now states: the L3 escalation signal is LIVE state from a /system endpoint (never a parsed dash-node.log line, which persists / a restore can copy in); firehoseStreak stays in-memory ON PURPOSE (resets on restart → self- freshens); and the app RE-PROBES to confirm the firehose is happening now before any destructive action — never acting on the report alone (ADR-5343's stale-state rule at runtime). Also comment log-rotate.ts to mark the warn diagnostic-only, not the signal.
… (observed on device) On-device testing surfaced a real edge: dash-node's 10-min guard timer resets on every restart, so anything that restarts it more often than the interval — a rebuild, a respawn loop, or a desired=DOWN vs pdsm-supervisor flap (a lingering WatchdogEnable=false from a prior Barrier-2 stop, in this case) — blinds the guard. It is the '(a clock-driven mechanism leaves gaps)' hazard made concrete. ADR-386 §8 records it, why the app-side disk-pressure backstop must stay the ultimate net, and why the Phase-2 pdsm-owned trigger (independent of dash-node's process lifetime) is the durable fix.
luisguzman-adfa
changed the title
This was referenced Sep 5, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
proot has no systemd and no running cron, so
/etc/cron.daily/logrotatenever runs —logrotate ships in the box but nothing triggers it, and a service log can grow until the
device hits ENOSPC (measured: php-fpm ~24 GB in one incident; dash-node/dash-rebuild logs
~1.8 GB each). This adds a K2Go-owned, proot-correct rotation config, a trigger, and a
firehose guard.
tools/setup-proot-logging.sh— idempotent configurator. Moves the inherited nginx/php-fpmsnippets out of
/etc/logrotate.d(we override them; a duplicate path fails logrotate) andinstalls
/etc/logrotate.d/k2go:copytruncate+size 100M+rotate 3+ compress forphp-fpm, nginx, calibre-web, dash-node/dash-rebuild. Validates with
logrotate -dand rollsback to the prior state on failure.
static/dashboard/sockets/log-rotate.ts— dash-node runs, every 10 min (no work at boot),a firehose guard then
logrotate:/var/log[/nginx]/*.logpast ~1 GiB in place first, so logrotatenever copies a multi-GB runaway (a copy doubles disk and pegs CPU on a weak phone). A firehose
log is garbage; a recurring one is flagged for the future app-side reap. Same timer as logrotate
(deterministic order, no drift).
updateStreaks(the recurrence logic) is pure + unit-tested.iiab-android(install_iiaboa_dashboard),tools/rebuild-dashboard.sh,tools/dev-push-dashboard.sheach run the configurator, so a clean rootfs ships preconfiguredand every update re-asserts it.
-h/--helpusage added to the four touched shell scripts (additive; existing args unchanged).1.2.12 → 1.3.0-dev.1(+ CHANGELOG) — not a REST-surface change, but the version bumpis how the new dash-node behavior reaches devices (the update pill; no ansible role yet). →
1.3.0at merge.
controller/docs/ADR-386— the umbrella design (prevent / contain logs / app-side backstop).Why proot-correct, not the inherited config
The inherited (Raspberry-Pi) snippets have no size cap and use postrotate signals (
invoke-rc.d,service reload) that fail under proot — a failed reopen leaves the daemon writing to the renamed
file (deleted-but-open).
copytruncatetruncates in place, so no signal is needed; size-basedrotation matches the threat (a file that grows, not a calendar). kiwix writes no dedicated log and
kolibri self-rotates — both left alone.
This is Layer 2 (contain logs) of ADR-386. It bounds steady/moderate growth; the guard keeps a
firehose from ever reaching logrotate. Stopping an off-proot orphan is the app-side backstop's job
(Layer 3, tracked on K2GO-386), and php-fpm-not-idle is Layer 1 (PR #543).
Verification
Device-verified on-device (OnePlus, dash-node 1.3.0-dev.1): a moderate log copytruncate-rotated
(150 MB →
.log.1, service kept running); a 1.17 GiB log was truncated in place by the guard on the10-min tick (
FIREHOSE: truncated … occurrence #1, file → 0, no.log.1— i.e. the guard, not alogrotate copy) with proot/nginx/node/kiwix all still up.
tsc --noEmit(strict) clean;updateStreaksunit tests 4/4; the four shell scripts pass
bash -n/sh -n.Config paths were verified on-device against the pdsm service wrappers and the live
/var/log.