K2GO-386 feat(app-backstop): app-side disk-fill backstop that keeps the system alive (L3b) - #550
Merged
Merged
Conversation
…he system alive (L3b) The app-side net for a disk-fill the in-box layers cannot stop (a service orphaned off proot; device-proven that only an app-side reap works). A background poller reads free space. On a CRITICAL reading DiskGuard does this: - Confirm with a fresh re-read after a short delay. It never acts on one reading. - Reap the box (EnvironmentProcess.reapBox) and reclaim the runaway log. - By DEFAULT keep the system alive. It does not force the server down. The ADFA-5343 reconciler is left desired=UP and asked to reconcile now, so it relaunches a fresh box (a fresh service under a fresh proot does not busy-loop). - Only when the fill returns 3 times inside 30 min, stop and stay down as a last resort and tell the user. - Report every trip to developers through the delivery backbone (enqueueAnalytics). This reorients the earlier stop-and-stay-down slice to restart-to-keep-alive, so a non-technical user is never left without a system for a transient fill. See ADR-386. Reuses the device-verified reap, the pure floor rule (DiskGuardPolicy, unit-tested), the StorageProbe read, the 25s poller, and the debug device-verify receiver.
…reap; fix debug/report Second code-review pass on L3b. Five fixes. 1. Deep-op guard. DiskGuard.deepOpActive(ctx) reads EnvironmentLock.currentHolder. If a STOPPED-class op (clone, backup, restore, install) holds the box, the tick logs and returns without reaping. A reap mid-extract would corrupt the rootfs. 2. Debug path no longer escalates. checkWithFloor runs in FORCED mode. FORCED always CONTAINs. It never advances the trip state. Before, three debug triggers could stop the box. 3. Consecutive-trip semantics. A non-critical tick ends the spell and resets the count. Escalation now means the disk stayed critical across consecutive restarts, so the restart is not fixing it. Before, the count reset only on a 30-min gap or app restart. 4. Testable escalation. The decision moves to the pure DiskGuardEscalation.next(...). No android.* imports. DiskGuardEscalationTest covers five cases on a plain JVM. 5. Report cadence. A CONTAIN reports only on the first trip of a spell. An ESCALATE always reports. A thrash no longer enqueues one analytics event per trip. See controller/docs/ADR-386-unattended-disk-containment.md.
…he reorient The debug hook doc still described the old stop-and-stay-down path (set desired=DOWN, box stays down). After the L3b reorient the debug hook runs the FORCED path: it always CONTAINs, reaps and reclaims, then leaves desired=UP and lets the reconciler relaunch the box. It never advances the escalation count. Correct the class Javadoc to match. Device-verified on HD1901.
…/K2GO-386-app-backstop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An app-side backstop (Layer 3 of ADR-386) that stops a runaway process from
filling the device disk, and keeps the system running while it does.
A background guard in
WatchdogServicepolls free space every 25 s. On aconfirmed CRITICAL reading it reaps the box, reclaims the runaway log, and by
default relaunches a fresh box through the ADFA-5343 reconciler. It stops and
stays down only as a last resort.
Why
On an unattended device serving content to non-technical users, a runaway
(e.g. an orphaned php-fpm busy-loop, K2GO-386) can fill the disk in hours. The
in-box layers cannot stop an off-proot orphan: an in-box kill does not reach it
(device-proven). Only the app, outside the rootfs, can. Layer 2 (log rotation,
already merged) bounds the logs; this layer is the net for what L2 cannot stop.
Stopping the box and leaving it down would deny service to a user who cannot fix
it. So the default is reap + restart, not stop-and-stay-down: a fresh service
under a fresh proot does not busy-loop.
How
DiskGuardPolicy(pure): the free-space floor and the CRITICAL rule.DiskGuardEscalation(pure): consecutive-trip semantics. A non-critical tickends the spell and resets the count. Escalation (stop and stay down) fires only
after the disk stays critical across several restarts in a row -- the restart is
not fixing it. Both rules are unit-tested on a plain JVM (no emulator).
DiskGuard: confirm-before-acting (a second read after a short delay, so itnever acts on one reading); skips the reap while a deep op (clone, backup,
restore, install) holds the box, so a reap cannot corrupt it; reclaims the
biggest runaway
.logover 1 GiB; reports each action to developers through thedelivery backbone; notifies the user only on the last-resort stop.
EnvironmentProcess.reapBox: reap the box and its services by cmdline token.src/debug) forces one check with an injected floor, toverify the path on device without filling the disk. It never ships in release.
New notification strings are in
strings_untranslated.xml(the tracking file),per the l10n convention.
Verification
DiskGuardPolicyTest,DiskGuardEscalationTest(green).4.2 GiB runaway log, freed the disk, relaunched a fresh box (new PIDs), and
enqueued the report. The debug hook never advances the real escalation count.
Coordination
this branch and lands in a follow-up PR (
feat/K2GO-386-firehose-signal).