diff --git a/controller/docs/ADR-386-unattended-disk-containment.md b/controller/docs/ADR-386-unattended-disk-containment.md new file mode 100644 index 000000000..51c830c59 --- /dev/null +++ b/controller/docs/ADR-386-unattended-disk-containment.md @@ -0,0 +1,217 @@ +# ADR-386 — Unattended disk-fill containment & maintainability (prevent · contain logs · reap-restart from the app) + +**Status:** proposed — design only, no production code. Lands as small diffs a maintainer verifies. This +is the single home for the K2GO-386 disk-fill work; if a layer's detail grows it splits into a lettered +delta (`ADR-386a`, `ADR-386b`) rather than a separate ADR. + +**Scope note (form):** genericized per the ADR authoring convention — no personal names, no device +identifiers. "the box" = the Debian userland under proot; "the device" = the Android target; "upstream" += Internet-in-a-Box (iiab/iiab). + +--- + +## 1. Why this exists (the stakes, stated plainly) + +A single misbehaving box process can **saturate the device's disk in hours** — faster on a faster phone +— until it hits `ENOSPC`. When that happens the server breaks *and the user's phone is left full and +unusable for anything else*. We have measured this class directly: an orphaned `php-fpm` busy-looping +into its log at ~600 MB–1.3 GB/min, and unbounded service logs reaching ~1.8 GB each with nothing ever +trimming them. + +Our users are **not technical**. The product's promise is a server that just works on a phone. So the +goal of this effort is not to detect-and-blame; it is to make the system **robust and simple to +administer**: + +- **resolve these situations unattended**, through **parameters** (thresholds/intervals we can tune), not + through a human running commands on the box; +- **keep the system alive** — recover by containing or restarting, not by denying service and walking + away (a monitor that spends battery only to leave the phone stopped in a bad state is worse than none); +- **report to the developers** the situations that warrant attention, so we learn what happens in the + field without the user having to notice or act. + +Everything below serves that goal. This is a maintainability effort as much as a bug fix. + +## 2. The problem, as facts + +Three failure shapes, all the same root class (a box process consuming disk faster than anything reclaims +it), plus the environment that makes them hard: + +- **Vector A — a runaway *real* log file:** visible on disk, grows without bound. +- **Vector B — deleted-but-open:** the process holds an *unlinked* fd; disk is consumed but there is **no + file** to find or rotate. Only **closing the fd (stop/restart the process)** frees it. +- **Speed:** the php busy-loop is a *firehose* (≫ any gentle rotation interval); most logs are a slow drip. + +Environment facts that shape every decision: + +- **proot has no systemd and no running cron** → `/etc/cron.daily/logrotate` **never runs**; logrotate is + installed but never triggered (root cause of the unbounded logs). +- **Upstream configs assume a Raspberry Pi** (systemd, always-on, cron, signal-based `postrotate`). Well + made for that world; unusable in ours. +- **An orphaned service is off proot's ptrace**, so an in-box healer cannot reach it — device-proven, only + an **outside-the-rootfs** actor (the Android app) can stop it. This is why the last layer lives in the app. + +## 3. The strategy — three layers, one home + +| Layer | Owns | Failure it handles | Where it runs | +|---|---|---|---| +| **L1 Prevent** | php-fpm does not run idle | removes the *cause* of the known orphan-loop on a default build | rootfs (ansible/patch) | +| **L2 Contain logs** | bounded, proot-correct rotation | Vector A, steady/moderate growth | in-box (dash-node-triggered) | +| **L3 App backstop** | reap + **restart to keep alive**, and report | Vector B + the fast firehose; the net for anything L2 misses | Android app (outside rootfs) | + +They are complementary, not redundant: L1 stops one cause, L2 bounds ordinary growth cheaply, L3 is the +outside net for the acute and the invisible. Reporting (§7) spans all three. + +## 4. Layer 1 — Prevent (php-fpm not idle) + +The idle `php-fpm` enabled unconditionally by the nginx role is the orphan that busy-loops. We move +php-fpm ownership to the roles that use it (Matomo enables its own), so a default build never runs it → +it cannot orphan. Carried as `tools/upstream-patches/0002-php-fpm-role-ownership.patch` (WIP upstream); +PR #543. **Note:** the rootfs build sources patches from `main`, so this takes effect only once merged +to `main` — a branch bake still reads `main`'s patches (verified). + +## 5. Layer 2 — Contain logs (K2Go owns its proot logging) + +We **own a centralized, proot-correct log-rotation config set** for the services we run, discarding the +RPi-oriented inherited snippets (learn from them, write ours): + +- **`copytruncate`, never a postrotate signal** — proot cannot drive `systemctl`/`invoke-rc.d`, and a + failed reopen is exactly Vector B. copytruncate truncates in place; the writer keeps its fd. +- **Size-based** (`size 100M`, `rotate 3`, `compress`, `delaycompress`, `missingok`, `notifempty`, + `su root root`) — our threat is a file that grows, not a calendar. +- **Trigger = dash-node** (the box's always-up process), **every 10 min, no run at boot** — boot is the + heaviest/most fragile moment (Python services starting; phantom-process-killer risk). Accepted tradeoff: + a <10-min session never rotates; the size cap + the guard below bound it. +- **A firehose guard runs FIRST, in the same tick** (deterministic order, one clock — no second timer to + drift): before logrotate, dash-node truncates any log past a firehose threshold (~1 GiB) **in place**, + so logrotate never has to copy a multi-GB runaway (a copy doubles disk and pegs CPU on a weak phone, and + still would not stop the writer). This is the in-box half of L3 (§6); a recurring firehose is flagged for + the app-side reap. + +**The config set (verified on-device against the pdsm wrappers + live `/var/log`):** + +| Service | Log | In the set | +|---|---|---| +| php-fpm | `/var/log/php8.4-fpm.log` | **yes** — override inherited snippet (add size cap, copytruncate) | +| nginx | `/var/log/nginx/*.log` | **yes** — override (drop `invoke-rc.d`, copytruncate) | +| calibre-web | `/var/log/calibre-web.log` (pdsm redirects here) | **yes** — added (`missingok`) | +| dash-node | `/var/log/dash-node.log`, `/var/log/dash-rebuild.log` | **yes** — added | +| kiwix | — none (`kiwix-serve --daemon`, no redirection) | **excluded** (no file to rotate) | +| kolibri | `/library/kolibri/logs/*.txt` (+ `archive/`) | **excluded** (self-managed, bounded ~64 KB) | + +**Install:** an idempotent `tools/setup-proot-logging.sh` (ansible-role-shaped) moves the inherited +nginx/php-fpm snippets out of `/etc/logrotate.d` (we override them; a duplicate path fails logrotate), +writes `/etc/logrotate.d/k2go`, and validates with `logrotate -d`. It runs **at deploy time, not at +dash-node startup** — reconfiguring on every boot would be needless churn (and updating the config is +exactly a deploy concern). The three deploy paths call it: the **rootfs build** (`iiab-android`'s +`install_iiaboa_dashboard`), so a clean R2 rootfs ships preconfigured; and the two update paths +(`rebuild-dashboard.sh`, `dev-push-dashboard.sh`). Since the version bump that carries a dash-node +change is delivered through the dashboard-update mechanism, an update that reaches a device also +re-asserts this config. Phase 2 folds it into a rootfs ansible role with a pdsm-owned trigger. + +## 6. Layer 3 — App-side backstop (reap + restart, not deny) + +The outside-the-rootfs net for what L2 cannot catch: the **fast firehose** (fills faster than a 10-min +rotation) and **Vector B** (no file to rotate; must stop the holding process). Device-proven: only the +app can stop an off-proot orphan. + +**L3 splits into two halves at the proot boundary — this matters:** +- **In-box (dash-node) — DETECT + RECLAIM.** dash-node can *see* a firehose log and *truncate* it in + place (reclaim, no copy) — but it **cannot stop an off-proot orphan** (an in-box kill does not reach it, + device-proven; the same limitation that parked the K2GO-381 in-box healer). This half is already coupled + to L2's tick as the firehose guard (§5): it protects L2 and bounds the disk (each tick truncates the + runaway back, so even an unstoppable orphan cannot reach ENOSPC as long as headroom > rate×interval), + and it emits the recurring-firehose signal. +- **App-side (Android) — STOP.** The only actor that can reap an off-proot orphan. It is driven by the + in-box recurring-firehose signal (a log that keeps refilling after truncation = an orphan) or by disk + pressure, and it reaps + reports. This is the half that is still to be built. + +**Decisions (direction; detailed mechanism is the next design step, possibly `ADR-386a`):** + +- **Targeted, not blind.** Act on the offending vector (a specific runaway log/service), keeping the rest + of the system up. NOT the rejected "disk full → stop everything → stay down → user fixes it." +- **Restart to keep alive.** A fresh service under a fresh proot does not busy-loop, so the recovery is + **reap + reclaim + relaunch through the lifecycle owner** (the ADR-5343 reconciler / desired-state), so + the box comes back — unattended. Stopping-and-staying-down is a last resort only, never the default. +- **Parametric.** Thresholds and cadence are parameters we tune (floor, growth-rate, intervals), not + hard-coded cliffs; the primary directive right now is **disk free space / abnormal growth**. +- **Single surface.** Watching disk pressure / abnormal log growth catches *any* vector, not just php. +- **Confirm before acting — the signal is a HINT, never a command (file the clock's edges).** Because the + channel is **pull, not push** (dash-node cannot call the app; verified — Express loopback, socket.io + retired), and every layer here is clock-driven, any reported state can be a tick — or a whole app + restart — **stale** by the time the app reads and acts on it. So: + - **The escalation signal is LIVE state, not a logged line.** The app reads dash-node's *current* + in-memory firehose state from a `/system/...` endpoint — it must **never parse a `[FIREHOSE]` line out + of `dash-node.log`**, because a log persists: an old line, or a log copied in by a restore, would fire a + false alarm. dash-node's `firehoseStreak` is **in-memory on purpose** — it resets when dash-node (or the + box) restarts, so the exact "user feels heat → closes the app → reopens it, and the firehose is now + gone" case comes back **clean**, with no stale state to act on. + - **The app re-probes before any destructive action.** On reading the hint it runs a **fresh check** — + is the firehose happening *right now* (disk pressure now, a log growing fast now)? — and reaps only on + confirmation. It never acts on the report alone. (ADR-5343's rule — do not act on stale/unowned state — + at runtime; the same "verify, don't suppose" we apply to code, applied to live state.) + - **The signal carries a timestamp; a stale one is ignored.** Freshness is part of the contract, not an + assumption. + +**Open (design next):** the exact leading-indicator (growth-rate vs absolute), the targeted-vs-full +decision, and how relaunch coordinates with the reconciler. The existing `feat/K2GO-386-disk-guard` +slice (device-verified) is the scaffold to reorient from "stop + stay-down" to this. + +## 7. Reporting to developers (unattended, cadence-based) + +When any layer contains or recovers a situation, it should tell us — so field robustness is measured, not +guessed — **without bothering the user**: + +- **Optional and cadence/rate-based:** a rare anomaly → a low-cadence digest (daily/weekly/monthly); a + recurring one (e.g. hourly) → escalate ("we're noticing X; send a report?"). A frequency/rate rule sets + the cadence. +- **Reuse the delivery backbone** (`DeliveryManager` / the debug-delivery path) rather than a new channel. +- The user is *informed*, not *tasked*: a report goes to developers; recovery already happened. + +## 8. Lifecycle (who sets it, who clears it, what if a process dies) + +- **L2 config:** installed at deploy time (rootfs build + rebuild/dev-push), idempotent — writes only + when changed, the snippet move is a no-op once done; re-asserted on every update, not every boot. No + persistent marker to strand. +- **L2 trigger:** dash-node's 10-min timer; owns nothing but the timer. If dash-node dies, logs only grow + while services are up and the reconciler owns bring-up; if dash-node *wedges*, rotation stalls — the one + Phase-1 liveness dependency, bounded by L3, removed in Phase 2 (pdsm-owned trigger). + - **Edge (observed on-device 2026-09-05): the timer resets on every dash-node restart.** The interval + fires only after dash-node runs *uninterrupted* for the full interval, so anything that restarts it more + often than 10 min — a rebuild, a crash/respawn loop, or a desired=DOWN vs pdsm-supervisor flap (which is + what a lingering `WatchdogEnable=false` from a prior Barrier-2 stop caused in testing) — **blinds the + guard**. This is exactly the "a clock-driven mechanism leaves gaps" hazard (§6). Normal operation runs + dash-node for hours, so it fires; but it is why (a) the disk-pressure app-side backstop must remain the + ultimate net if the in-box guard is starved, and (b) the Phase-2 pdsm-owned trigger (a supervisor loop + independent of dash-node's process lifetime) is the durable fix. +- **L3:** a poller for the life of a box-up session (started once, stopped on teardown); recovery routes + through the ADR-5343 desired-state owner, so no second source of "should the box be up." +- **State:** logrotate's own status file + `rotate N`/`compress` bound disk; nothing we add persists + unbounded. + +## 9. Forks considered and rejected + +- **Run cron/crond in the box** — Phase 1 rejects it (dash-node can schedule); revisit only for a + dash-node-independent trigger (Phase 2 pdsm). +- **Keep upstream snippets, just trigger them** — rejected: no size caps + proot-broken postrotate signals + (re-creates Vector B). +- **A light logrotate pass at boot** — rejected for now (boot lightness wins; cap + L3 bound the gap). +- **Bake L2 into the rootfs now** — deferred to Phase 2, not rejected. +- **L3 = stop-and-stay-down** — rejected: denies service to a non-technical user (the whole point of §1). + +## 10. Verification (per layer) + +| Layer | Check | Expected | +|---|---|---| +| L1 | rootfs built with the merged patch | php-fpm installed, **not enabled**, not running on a default build | +| L2 | `logrotate -d` after install; a log grown past `size`, then a trigger | parses clean; truncated in place; writer keeps writing; no orphaned deleted-but-open file | +| L2 | short (<10 min) session | no boot rotation (by design); bounded next long session | +| L3 | fast fill + Vector B (synthetic) | contained/recovered without denying service; system back up; a report enqueued | + +## 11. Consequences + +- Disk-fill is handled unattended across its vectors, and the phone is kept usable — matching §1. +- We carry a small owned config set instead of inherited RPi drift; new services get one block, one place. +- One liveness dependency remains in Phase 1 (dash-node as L2 trigger); named, bounded by L3, removed in + Phase 2. +- Field occurrences are reported to developers, so we tune the parameters from real data rather than + guesses. diff --git a/iiab-android b/iiab-android index faffc1478..10616af16 100644 --- a/iiab-android +++ b/iiab-android @@ -19,6 +19,17 @@ log() { printf "${BLU}[iiab]${RST} %s\n" "$*"; } warn() { printf "${YEL}[iiab] WARNING:${RST} %s\n" "$*" >&2; } die() { printf "${RED}[!] ERROR${RST}: %s\n" "$*" >&2; exit 1; } +usage() { + cat <<'USAGE' +iiab-android — K2Go rootfs installer. Runs the IIAB install under proot and installs the K2Go +dashboard, services, content, and log rotation. Run as ROOT inside the proot box; normally invoked +by tools/rootfs-builder/build-iiab-rootfs.sh, not by hand. + +Usage: iiab-android [-h|--help] +USAGE +} +case "${1:-}" in -h|--help) usage; exit 0 ;; esac + #----------------------------- # Safety checks #----------------------------- @@ -512,6 +523,11 @@ install_iiaboa_dashboard() { /usr/local/bin/pdsm enable dash-node /usr/local/bin/pdsm restart dash-node + # K2GO-386 (ADR-386): install K2Go-owned log rotation for the box (proot has no cron, so a service + # log can otherwise fill the device). Run at deploy, not at dash-node boot. Non-fatal. + log "Configuring log rotation (setup-proot-logging)..." + sh "${K2GO_OPT_DIR}/tools/setup-proot-logging.sh" || warn "log-rotation setup failed (non-fatal)" + # Setup nginx log "Configuring Nginx for the dashboard..." cp "${dash_src}/dash-node-nginx.conf" "/etc/nginx/conf.d/dash-node-nginx.conf" diff --git a/static/dashboard/CHANGELOG.md b/static/dashboard/CHANGELOG.md index 72fc339ee..d10aaa820 100644 --- a/static/dashboard/CHANGELOG.md +++ b/static/dashboard/CHANGELOG.md @@ -4,6 +4,7 @@ One line per version, newest first. Every REST-facing change bumps the version i (the app surfaces it via `/system/dashboard/update-check` and the "Update available" pill), so this file is the human record of what each bump enables. Keep entries short: `version - change (TICKET)`. +- **1.3.0-dev.1** - Proot log rotation, dash-node-triggered (K2GO-386, ADR-386). proot has no systemd/cron, so `/etc/cron.daily/logrotate` never runs — logrotate was installed but never triggered, and a service log (php-fpm, dash-node) could grow until the device hit ENOSPC. dash-node now runs, every 10 min (no work at boot; `timer.unref`), a firehose guard THEN `logrotate /etc/logrotate.conf`: the guard truncates any log past ~1 GiB in place first (a runaway ~GB/min that logrotate would otherwise copy — doubling disk + pegging CPU on a weak phone), so L2 never meets a firehose; a recurring firehose is flagged for the future app-side reap (ADR-386 §6). The K2Go-owned config `/etc/logrotate.d/k2go` (copytruncate + `size 100M`, proot-correct — no reopen signal — overriding the RPi-oriented nginx/php-fpm snippets and adding calibre-web + dash-node; kiwix has no log, kolibri self-rotates) is installed at deploy by `tools/setup-proot-logging.sh` (rootfs build + rebuild/dev-push), not at boot. Not a REST-surface change; the version bump is the delivery mechanism for the new dash-node behavior (no ansible role yet). WIP → 1.3.0 at merge. (K2GO-386) - **1.2.12** - Dashboard-update card back end (ADFA-5339, Phase 1 server half). New read-only `GET /system/dashboard/rebuild/log`: the last ~200 lines of `/var/log/dash-rebuild.log`, for the card's expandable "Details" (no file yet = empty log, not an error). `POST /system/dashboard/rebuild` now accepts `{ site: true }`: it refreshes the served landing page in the SAME run via `site-updater.sh`, from the same clone the rebuild's git fetch+reset refreshes, in finalize AFTER the core swap verifies live — so the site matches the new source. The site is a separate, versionless artifact: it never touches the reported version, and a site failure is logged, never a rollback of the (already-verified) core update. Both localhost-only. (ADFA-5339) - **1.2.11** - `/auth/:service/session` mints the session **for the agent that asks** (ADFA-5361). Calibre-Web (Flask-Login) binds a session to a fingerprint of the User-Agent, so a session minted under dash-node's own agent was rejected on the WebView's first request: the identity was dropped, the `remember_token` deleted, and the card opened as the anonymous Guest — the "logged in as Admin" flash comes from the injected session and renders even then, which is why the auto-login looked like it worked. The route now forwards the caller's `User-Agent` through the whole login handshake (every request, not just the POST — the fingerprint is established on the first one), for Calibre-Web and Kolibri alike. The callers that consume the session themselves (downloads runner, `removeBook`) are unchanged. No User-Agent on the request degrades to the previous behaviour, logged. Same ticket: the books runner's private copy of the Calibre-Web login is gone — it never got the ADFA-5043 `remember_me` and was the drift this whole bug rode in on — so `getCalibreSession` is the one source. (ADFA-5361) - **1.2.10** - In-proot content-service recovery (ADFA-5343, ADR-5343a §10). New `POST /system/service/:svc/restart`: runs `pdsm restart ` in the one living proot to recover a content service wedged after an environment relaunch (orphaned off proot → `epoll_wait` ENOSYS), for the supported upstream services (mirrors `pdsm_installed_services`; `dash-node` excluded). A server-side watcher auto-heals a present-but-wedged content service (404 = not installed → left alone), cooldown-bounded; the app's future module-card Retry is the manual backstop hitting the same endpoint. Loopback-only, like all of `/k2go-api`. (ADFA-5343) diff --git a/static/dashboard/package.json b/static/dashboard/package.json index 03c376746..fc4442b24 100644 --- a/static/dashboard/package.json +++ b/static/dashboard/package.json @@ -1,10 +1,10 @@ { "name": "dashboard-console", - "version": "1.2.12", + "version": "1.3.0-dev.1", "description": "", "main": "index.js", "scripts": { - "test": "node --require ts-node/register --test sockets/maps.socket.test.ts sockets/rolling-log.test.ts sockets/kolibri.session.test.ts sockets/credentials.test.ts sockets/net-retry.test.ts sockets/services.test.ts", + "test": "node --require ts-node/register --test sockets/maps.socket.test.ts sockets/rolling-log.test.ts sockets/kolibri.session.test.ts sockets/credentials.test.ts sockets/net-retry.test.ts sockets/services.test.ts sockets/log-rotate.test.ts", "test:db": "node --require ts-node/register --test sockets/jobs.test.ts", "typecheck": "tsc --noEmit", "build": "tsc", diff --git a/static/dashboard/server.ts b/static/dashboard/server.ts index feac9f5c9..87fe409d0 100644 --- a/static/dashboard/server.ts +++ b/static/dashboard/server.ts @@ -11,6 +11,7 @@ import './sockets/books.exec'; import './sockets/kolibri.exec'; import { apiRouter } from './routes'; import { startServiceHeal } from './sockets/service-heal'; +import { startLogRotation, stopLogRotation } from './sockets/log-rotate'; const app = express(); const server = http.createServer(app); @@ -40,6 +41,10 @@ server.listen(PORT, '127.0.0.1', () => { try { jobs.reconcileOnBoot(); } catch (e) { console.error('[jobs] reconcile failed', e); } // ADFA-5343 (ADR-5343a §10): the box heals its own content-service tree in-proot. try { startServiceHeal(); } catch (e) { console.error('[service-heal] start failed', e); } + // K2GO-386 (ADR-386 §4): proot has no cron — dash-node triggers logrotate on a timer so the + // K2Go-owned rotation config (installed at deploy time) actually runs (no rotation at boot; + // first pass at +10 min). + try { startLogRotation(); } catch (e) { console.error('[log-rotate] start failed', e); } }); // ========================================== @@ -48,6 +53,9 @@ server.listen(PORT, '127.0.0.1', () => { const gracefulShutdown = (signal: string) => { console.log(`\n[System] Received ${signal}. Starting graceful shutdown...`); + // K2GO-386: clear the log-rotation timer (who starts it: startLogRotation; who clears it: here). + try { stopLogRotation(); } catch (e) { console.error('[log-rotate] stop failed', e); } + server.close(() => { console.log('[System] HTTP server closed. No longer accepting connections.'); console.log('[System] Cleanup complete. Exiting safely.'); diff --git a/static/dashboard/sockets/log-rotate.test.ts b/static/dashboard/sockets/log-rotate.test.ts new file mode 100644 index 000000000..3d5b3b980 --- /dev/null +++ b/static/dashboard/sockets/log-rotate.test.ts @@ -0,0 +1,33 @@ +/// +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { updateStreaks } from './log-rotate'; + +test('updateStreaks: a first-time firehose starts at occurrence 1', () => { + const next = updateStreaks(new Set(['/var/log/php8.4-fpm.log']), new Map()); + assert.equal(next.get('/var/log/php8.4-fpm.log'), 1); +}); + +test('updateStreaks: a recurring firehose increments from its previous count', () => { + const prev = new Map([['/var/log/php8.4-fpm.log', 2]]); + const next = updateStreaks(new Set(['/var/log/php8.4-fpm.log']), prev); + assert.equal(next.get('/var/log/php8.4-fpm.log'), 3); +}); + +test('updateStreaks: a path not firehosing this pass is dropped (no stale/leaked entry)', () => { + // gone.log firehosed before but is not in this pass (deleted, renamed, or now sane) → must not linger. + const prev = new Map([ + ['/var/log/gone.log', 5], + ['/var/log/dash-node.log', 1], + ]); + const next = updateStreaks(new Set(['/var/log/dash-node.log']), prev); + assert.equal(next.has('/var/log/gone.log'), false); + assert.equal(next.get('/var/log/dash-node.log'), 2); + assert.equal(next.size, 1); +}); + +test('updateStreaks: an empty firehose set clears everything (a calm tick leaks nothing)', () => { + const prev = new Map([['/var/log/a.log', 3]]); + const next = updateStreaks(new Set(), prev); + assert.equal(next.size, 0); +}); diff --git a/static/dashboard/sockets/log-rotate.ts b/static/dashboard/sockets/log-rotate.ts new file mode 100644 index 000000000..d499299eb --- /dev/null +++ b/static/dashboard/sockets/log-rotate.ts @@ -0,0 +1,118 @@ +// sockets/log-rotate.ts — K2GO-386 / ADR-386 (Layer 2: contain logs) +// +// proot has no systemd/cron, so nothing runs logrotate on its own. dash-node — the box's always-up +// process — drives log containment on a fixed timer. Each tick, IN ORDER: +// 1. firehose GUARD: truncate any pathologically huge .log IN PLACE, before logrotate can try to +// copy it. logrotate's copytruncate would COPY a 6 GB runaway log (doubling disk, pegging CPU on +// a weak phone) and still not stop the writer — so we reclaim it in seco first (no copy, no gzip). +// A firehose log is garbage (repeated error spam); we discard it. This is the in-box half of L3 +// (ADR-386 §6): dash-node can DETECT + RECLAIM, but it CANNOT stop an off-proot orphan (device- +// proven — an in-box kill does not reach it); STOPPING that is app-side. A recurring firehose is +// flagged for that app-side reap. +// 2. logrotate: rotate the remaining MODERATE logs (copytruncate + size-based, cheap now). +// Coupling the guard to the SAME timer as logrotate gives a deterministic order (guard→rotate) with +// ONE clock — so L2 never meets a firehose, and there is no second timer to drift out of sync. +// +// The config itself is INSTALLED at deploy time (tools/setup-proot-logging.sh), not here and not at +// boot. There is NO work at boot either (setInterval fires first at +interval): boot is the heaviest, +// most fragile moment under proot and we keep it clear. Everything here is best-effort. +import { execFile } from 'child_process'; +import * as fs from 'fs'; +import * as path from 'path'; + +// Rotate every 10 min (ADR-386 §4). Wide enough to be nearly free; narrow enough to bound growth. +const INTERVAL_MS = 10 * 60 * 1000; + +// logrotate lives in /usr/sbin, which dash-node's runtime PATH (set by its pdsm wrapper) may not +// include — call it by absolute path so a reduced PATH can't turn every tick into an ENOENT. +const LOGROTATE_BIN = '/usr/sbin/logrotate'; + +// A .log past this size is a firehose (a runaway ~GB/min), not a normal log (which are KB–MB). logrotate +// would copy it; we truncate it in place instead. Parametric — tune per device headroom / interval. +const FIREHOSE_BYTES = 1024 * 1024 * 1024; // 1 GiB +// The dirs logrotate's k2go config covers; the guard scans the same surface for *.log files. +const LOG_DIRS = ['/var/log', '/var/log/nginx']; + +// Consecutive-truncation count per path, so a RECURRING firehose (an orphan the box cannot stop, that +// just refills) is flagged for the app-side reap (ADR-386 §6). In-memory ON PURPOSE: it resets when +// dash-node (or the box) restarts, so a resolved-by-restart firehose comes back clean with no stale +// state (ADR-386 §6, "confirm before acting"). The console.warn below is DIAGNOSTIC/historical — the +// L3 escalation signal must be LIVE state read from a /system endpoint, never a parsed dash-node.log line. +// Only paths firehosing this pass carry forward (see updateStreaks), so a deleted/renamed log never lingers. +let firehoseStreak = new Map(); + +/** Pure: the updated streak counts given the paths firehosing THIS pass and the previous counts — each + * firehosing path +1, everything else dropped (so nothing leaks when a log disappears). No I/O; + * unit-tested (log-rotate.test.ts). */ +export function updateStreaks(firehosing: Set, prev: Map): Map { + const next = new Map(); + for (const p of firehosing) next.set(p, (prev.get(p) || 0) + 1); + return next; +} + +let timer: NodeJS.Timeout | null = null; + +/** Pre-logrotate guard: truncate any firehose-sized .log IN PLACE so logrotate never has to copy it. + * Reclaims instantly (no copy, no compress). Best-effort per file. */ +function guardFirehoseLogs(): void { + const firehosing: Array<{ p: string; size: number }> = []; + for (const dir of LOG_DIRS) { + let entries: string[]; + try { entries = fs.readdirSync(dir); } catch { continue; } + for (const name of entries) { + if (!name.endsWith('.log')) continue; + const p = path.join(dir, name); + try { + const size = fs.statSync(p).size; + if (size <= FIREHOSE_BYTES) continue; + fs.truncateSync(p, 0); // reclaim in place — no copy, no compress + firehosing.push({ p, size }); + } catch { /* best-effort per file */ } + } + } + // Update the streaks in one pass: only paths that firehosed now carry forward (no stale entries). + firehoseStreak = updateStreaks(new Set(firehosing.map((f) => f.p)), firehoseStreak); + for (const { p, size } of firehosing) { + const n = firehoseStreak.get(p) || 1; + console.warn( + `[log-rotate] FIREHOSE: truncated ${p} (${size} B) in place, occurrence #${n}` + + (n >= 2 ? ' — recurring; likely an off-proot orphan, needs app-side reap (ADR-386 L3)' : ''), + ); + } +} + +/** Run one logrotate pass over the whole config (the K2Go blocks use copytruncate + size, so this is + * cheap unless a file actually exceeds its cap). Best-effort. */ +function runLogrotateOnce(): void { + execFile(LOGROTATE_BIN, ['/etc/logrotate.conf'], { timeout: 60_000 }, (err, _stdout, stderr) => { + if (err) { + console.error('[log-rotate] logrotate run failed:', err.message, (stderr || '').trim()); + } + }); +} + +/** One tick: the firehose guard first (so L2 never meets a firehose), then logrotate the moderate logs. */ +function tick(): void { + try { guardFirehoseLogs(); } catch (e) { console.error('[log-rotate] firehose guard failed', e); } + runLogrotateOnce(); +} + +/** + * Start the periodic log-containment trigger. Idempotent; call once from server.ts at listen time. + * Runs guard+logrotate every INTERVAL_MS — the FIRST run is at +INTERVAL_MS, never at boot (ADR-386 §4). + */ +export function startLogRotation(): void { + if (timer) return; + timer = setInterval(tick, INTERVAL_MS); + if (typeof timer.unref === 'function') timer.unref(); // don't keep the process alive just for this + console.log( + `[log-rotate] scheduled: firehose guard + logrotate every ${INTERVAL_MS / 60000} min (none at boot)`, + ); +} + +export function stopLogRotation(): void { + if (timer) { + clearInterval(timer); + timer = null; + } +} diff --git a/tools/dev-push-dashboard.sh b/tools/dev-push-dashboard.sh index acc2564ed..804cce993 100755 --- a/tools/dev-push-dashboard.sh +++ b/tools/dev-push-dashboard.sh @@ -17,6 +17,20 @@ # deploy dash-node-nginx.conf to /etc/nginx/conf.d (nginx does not read /library/dashboard). set -eu +usage() { + cat <<'USAGE' +dev-push-dashboard.sh — push an updated dashboard from a local clone into the INSTALLED rootfs and +restart the service, WITHOUT the ~2h rootfs rebuild (ADFA-4839). Run from INSIDE the proot. + +Usage: sh dev-push-dashboard.sh [CLONE_DIR] + CLONE_DIR clone to deploy from (default: the repo this script lives in) + +Syncs static/dashboard -> /library/dashboard (preserving node_modules), installs log rotation, +deploys the nginx vhost, rebuilds (yarn build), and restarts dash-node + nginx. +USAGE +} +case "${1:-}" in -h|--help) usage; exit 0 ;; esac + CLONE_DIR="${1:-$(cd "$(dirname "$0")/.." && pwd)}" SRC="$CLONE_DIR/static/dashboard" DEST="/library/dashboard" @@ -43,6 +57,9 @@ echo "[dev-push] deploying nginx vhost to $NGINX_CONF_DIR..." cp -f "$DEST/dash-node-nginx.conf" "$NGINX_CONF_DIR/dash-node-nginx.conf" chmod 0600 "$NGINX_CONF_DIR/dash-node-nginx.conf" +echo "[dev-push] configuring log rotation (setup-proot-logging)..." +sh "$CLONE_DIR/tools/setup-proot-logging.sh" || echo "[dev-push] warn: log-rotation setup failed (non-fatal)" + echo "[dev-push] restarting dash-node + nginx..." /usr/local/bin/pdsm restart dash-node /usr/local/bin/pdsm restart nginx diff --git a/tools/rebuild-dashboard.sh b/tools/rebuild-dashboard.sh index 04fb586b0..850773bd5 100755 --- a/tools/rebuild-dashboard.sh +++ b/tools/rebuild-dashboard.sh @@ -26,6 +26,21 @@ set -u # is how the app's detached REST call passes it). CLONE_DIR is the optional $2 — the install location is # almost always the same, so it defaults; pass it only if it moved. # sh tools/rebuild-dashboard.sh [clone_dir] +usage() { + cat <<'USAGE' +rebuild-dashboard.sh — rebuild ONLY the dash-node REST API from the on-device clone, no rootfs +rebuild. Blue-green + verify-before-commit (ADFA-5011). + +Usage: sh rebuild-dashboard.sh [BRANCH] [CLONE_DIR] + BRANCH git branch to fetch + reset --hard from origin (default: $K2GO_BRANCH, else main) + CLONE_DIR clone location to build from (default: /opt/iiab-android) + +Run inside the proot box. Fetches origin/, builds in a staging dir, smoke-tests it, +atomically swaps the dist in, restarts dash-node, and verifies live (rolls back on failure). +USAGE +} +case "${1:-}" in -h|--help) usage; exit 0 ;; esac + BRANCH="${1:-${K2GO_BRANCH:-main}}" CLONE_DIR="${2:-/opt/iiab-android}" SRC="$CLONE_DIR/static/dashboard" @@ -165,6 +180,8 @@ if verify_live; then # nginx reads /etc/nginx/conf.d, not /library/dashboard, so mirror the vhost then reload nginx. [ -f "$LIVE/dash-node-nginx.conf" ] && { cp -f "$LIVE/dash-node-nginx.conf" "$NGINX_CONF_DIR/dash-node-nginx.conf"; chmod 0600 "$NGINX_CONF_DIR/dash-node-nginx.conf"; } /usr/local/bin/pdsm restart nginx >>"$LOG" 2>&1 || log "warn: pdsm restart nginx returned non-zero" + # K2GO-386 (ADR-386): re-assert K2Go-owned log rotation on every update (proot has no cron). + sh "$CLONE_DIR/tools/setup-proot-logging.sh" >>"$LOG" 2>&1 || log "warn: log-rotation setup failed (non-fatal)" # ADFA-5339: optionally refresh the served landing page, from the SAME clone the git fetch+reset # above just refreshed, so it matches the new source. Runs only here — after the core swap has # verified live — and is best-effort: the site is a separate, versionless artifact, so a failure is diff --git a/tools/setup-proot-logging.sh b/tools/setup-proot-logging.sh new file mode 100644 index 000000000..f806b5797 --- /dev/null +++ b/tools/setup-proot-logging.sh @@ -0,0 +1,118 @@ +#!/bin/sh +# tools/setup-proot-logging.sh — K2GO-386 / ADR-386 (Layer 2: contain logs) +# +# Install K2Go's proot-correct log-rotation config for the box, idempotently. +# +# WHY: proot has no systemd and no running cron, so /etc/cron.daily/logrotate NEVER runs — logrotate +# is installed but nothing triggers it, and a service log can grow until the device hits ENOSPC. The +# inherited (Raspberry-Pi-oriented) snippets also lack size caps and use postrotate signals that fail +# under proot (a failed reopen = the deleted-but-open failure). We own our rotation instead. +# +# This script only INSTALLS the config; it does NOT rotate. The rotation itself is triggered on a +# timer by dash-node (static/dashboard/sockets/log-rotate.ts), the box's always-up process. Run this +# inside the proot box (where /etc/logrotate.d exists), at DEPLOY time (rootfs build + rebuild/dev-push). +# +# What it does (all idempotent): +# 1. Move the inherited nginx + php*-fpm snippets OUT of /etc/logrotate.d — we override them, and two +# snippets listing the same path make logrotate fail with "duplicate log entry". +# 2. Write /etc/logrotate.d/k2go (copytruncate + size-based; proot-correct — no reopen signal). +# 3. Validate the whole config with `logrotate -d` (parse only); on failure ROLL BACK to the prior +# state (never leave a broken config that would break ALL rotation) and fail loudly. +set -eu + +usage() { + cat <<'USAGE' +setup-proot-logging.sh — install K2Go-owned log rotation for the proot box (K2GO-386 / ADR-386). + +Usage: sh setup-proot-logging.sh [-h|--help] + +Takes no arguments. Run inside the proot box, at DEPLOY time (rootfs build + rebuild/dev-push). +It overrides the inherited nginx/php-fpm logrotate snippets, installs /etc/logrotate.d/k2go +(copytruncate + size-based), validates with `logrotate -d`, and rolls back on failure. Idempotent. +dash-node triggers the rotation itself on a 10-min timer; this script only installs the config. +USAGE +} +case "${1:-}" in -h|--help) usage; exit 0 ;; esac + +LR_D="/etc/logrotate.d" +OVERRIDDEN="/etc/logrotate.d.k2go-overridden" # OUTSIDE LR_D, so logrotate never reads it +K2GO_CONF="$LR_D/k2go" +# Resolve logrotate by absolute path (a deploy shell may have a reduced PATH without /usr/sbin). +LOGROTATE="$(command -v logrotate 2>/dev/null || echo /usr/sbin/logrotate)" + +[ -d "$LR_D" ] || { echo "[k2go-logging] $LR_D not found (not inside the box?) — nothing to do" >&2; exit 0; } +[ -x "$LOGROTATE" ] || { echo "[k2go-logging] logrotate not installed ($LOGROTATE) — skipping" >&2; exit 0; } + +# --- snapshot for rollback (restore EXACTLY the pre-run state on validation failure) -------------- +PREV_K2GO_BAK="" +if [ -f "$K2GO_CONF" ]; then PREV_K2GO_BAK="$(mktemp)"; cp -f "$K2GO_CONF" "$PREV_K2GO_BAK"; fi +MOVED="" # basenames this run moved aside, so rollback restores only those + +rollback() { + if [ -n "$PREV_K2GO_BAK" ]; then cp -f "$PREV_K2GO_BAK" "$K2GO_CONF"; else rm -f "$K2GO_CONF"; fi + for base in $MOVED; do mv -f "$OVERRIDDEN/$base" "$LR_D/$base" 2>/dev/null || true; done +} + +# 1) Override the inherited snippets for services we now own. nginx is a fixed name; php*-fpm is a +# glob so a php-version bump (php8.5-fpm) is covered too — it MUST match the config's php*-fpm.log +# glob below, or the un-moved snippet would collide with our block. +mkdir -p "$OVERRIDDEN" +for snip in "$LR_D/nginx" "$LR_D"/php*-fpm; do + [ -f "$snip" ] || continue + base="$(basename "$snip")" + mv -f "$snip" "$OVERRIDDEN/$base" + MOVED="$MOVED $base" + echo "[k2go-logging] overrode inherited snippet: $base (moved to $OVERRIDDEN)" +done + +# 2) Write our config, only if it changed (idempotent). +NEW="$(mktemp)" +cat > "$NEW" <<'EOF' +# K2GO-386 / ADR-386 — K2Go-owned log rotation for the proot box. DO NOT edit by hand; +# managed by tools/setup-proot-logging.sh. +# +# proot has no systemd/cron: dash-node triggers `logrotate` on a timer. Every block uses +# copytruncate (no daemon reopen signal — a failed reopen under proot is the deleted-but-open +# failure) and rotates on SIZE (our threat is a file that grows, not a schedule). This overrides +# the inherited nginx/php-fpm snippets (moved to /etc/logrotate.d.k2go-overridden) and adds the +# ones that never shipped (calibre-web, dash-node). kiwix writes no dedicated log; kolibri +# self-rotates under its KOLIBRI_HOME — both are deliberately left alone. +/var/log/php*-fpm.log +/var/log/nginx/*.log +/var/log/calibre-web.log +/var/log/dash-node.log +/var/log/dash-rebuild.log +{ + su root root + size 100M + rotate 3 + compress + delaycompress + missingok + notifempty + copytruncate +} +EOF + +if [ -f "$K2GO_CONF" ] && cmp -s "$NEW" "$K2GO_CONF"; then + echo "[k2go-logging] $K2GO_CONF already up to date" + rm -f "$NEW" +else + cp -f "$NEW" "$K2GO_CONF" + chmod 0644 "$K2GO_CONF" + rm -f "$NEW" + echo "[k2go-logging] installed $K2GO_CONF" +fi + +# 3) Validate the WHOLE effective config (parse only; does NOT rotate). On failure, roll back to the +# pre-run state so a broken config never breaks all rotation, then fail loudly. +if "$LOGROTATE" -d /etc/logrotate.conf >/dev/null 2>&1; then + echo "[k2go-logging] logrotate config validates OK" + rm -f "$PREV_K2GO_BAK" +else + echo "[k2go-logging] ERROR: logrotate config failed validation — rolling back:" >&2 + "$LOGROTATE" -d /etc/logrotate.conf 2>&1 | tail -20 >&2 + rollback + rm -f "$PREV_K2GO_BAK" + exit 1 +fi