ADFA-5343 feat(server-lifecycle): process-scoped reconciler (Phases 0–2) + flap auto-recovery + in-proot content recovery - #498
Merged
Conversation
…er execution class
The desired-state predicate used EnvironmentLock.currentHolder == NONE, forcing the
server DOWN for every holder. But LIVE holders run against the live server and need it
UP: DOWNLOAD (ZIM/Books/Kolibri are ExecutionClass.LIVE; the device only POSTs+polls,
ZimDownloadService.java:10) and DASHBOARD (live dash-node self-update). This also
contradicted the ADR's own section 6, which already requires DASHBOARD to stay UP.
Refine the predicate to key on the holder execution class:
desired = installed && healthy && userWantsOn
&& currentHolder.executionClass != STOPPED
Reuse the existing ADR-5061 type Operation.ExecutionClass { LIVE, STOPPED }
(Operation.java:49-61) as one property on the Holder enum (STOPPED for
CLONE/BACKUP/RESTORE/INSTALL; LIVE for DOWNLOAD/DASHBOARD/NONE) rather than adding a
parallel HolderClass. No new source of truth; NONE keeps desired=UP as before.
Design edit only; the Holder enum property lands with Phase 0 code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…erLiveness) Add env/domain/ServerLiveness, the single freshness-windowed liveness snapshot the reconciler effort collapses the four scattered sources into. Pure JVM: it holds the two observed facts (processPresent from /proc, servicesAnswering from /k2go-api) plus a monotonic observedAtMs, and derives one Phase (UNKNOWN/DOWN/STARTING/UP). Reuses Freshness.fresh for the one definition of "still trustworthy"; a never-observed or stale snapshot folds to UNKNOWN (absorbs hasObservation()). Route the 3s status poll through it (ServerController.checkServerStatus): build the snapshot from EnvironmentProcess + RestReadiness, publish alive = (phase == UP) to ServerStateRepository as before. The one behavior shift is that "up" now means the services answer (/k2go-api) rather than nginx (/home), which answers before its dash-node upstream is ready - so a flap no longer reads as spurious "up" (ADFA-5336). Downstream is otherwise unchanged: ServerState stays a 1-bit alive fact, no consumer touched. Delete the now-unused /home pingUrl and its two java.net imports. JVM tests cover the four-phase truth table and the freshness boundary (ServerLivenessTest). startEnvironment's own liveness reads are intentionally left for Phase 4; Phase 0 is additive. Verified: pure-domain javac compile + the 9 JVM tests pass offline. Full app compile + lint must run via Gradle; runtime (boot/up/down on Home; flap no longer shows spurious up) is device-only. Rollback: revert ServerController.java; the poll returns to the /home ping (the new ServerLiveness file is then unused). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ired-state Introduce the one process-scoped owner of "the server should be up", first as a LOG-ONLY observer (no actuation), plus the pure decisions it will own. - ServerReconcile (pure JVM): desired() = installed && healthy && userWantsOn && holderClass != STOPPED - a pure function of facts that already have owners, so no new source of truth. intent() = the coarse direction (START/STOP/WAIT/HOLD/NOOP) from desired vs the observed ServerLiveness.Phase; an UNKNOWN phase always HOLDs. - EnvironmentLock.Holder now carries its Operation.ExecutionClass (ADR-5061's one LIVE/STOPPED type, reused - not a parallel enum). This lands the approved Task-1 predicate fix in code: LIVE holders (download, dashboard self-update) leave desired UP; only STOPPED holders (clone/backup/restore/install) pull it down; NONE is LIVE. - ServerLifecycleReconciler (app-scoped singleton, established in IIABApplication): observe(ctx, liveness) computes desired from SystemFactsReader + WatchdogEnable + currentHolder and logs desired-vs-actual -> wouldDo each tick. No actuation. It is FED the Phase-0 snapshot by the status poll, so there is no second liveness source. JVM tests: ServerReconcileTest covers the desired truth table (the holder-class dimension in particular) and the intent table. Verified: :app:testDebugUnitTest + :app:lintDebug BUILD SUCCESSFUL locally. Device- only: logcat "K2Go-Reconciler" on all five flows (boot, module hand-off, deep ops, dashboard rebuild, turn-off) should show desired/actual/wouldDo matching reality with no surprises. Still additive - the scorecard reduction lands when Phase 4 deletes the scaffolding these parts replace. Rollback: delete ServerReconcile + ServerLifecycleReconciler and their two call sites (the ServerController poll seam and the IIABApplication warm-up); revert the Holder field. Nothing depends on the observer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…odule hand-off (5336) First actuation. The module-batch hand-off stops booting + polling the server itself and instead sets the desired state; the reconciler brings the box up and keeps it up, re-driving a post-install flap so the hand-off can no longer strand a dead Home (5336). Actuation via a bridge (A1), not a new boot path: - ServerLifecycleReconciler gains an Actuator interface. The foregrounded ServerController registers on onResume and clears on onPause (compare-and-clear, idempotent to a resume/pause overlap). The reconciler owns the DECISION (desired); the boot MECHANISM stays the one existing, idempotent path (ServerController.startEnvironment via EnvironmentEnsure). No second actuator, no double-proot. Off-UI boot + own tick are Phase 4. - On a tick, when desired=UP and the box is not confirmed up, the reconciler calls ensureServerUp() through the registered actuator. START (down) and WAIT (coming up / stuck flap) both route there (ServerReconcile.ensuresUp); startEnvironment's EnvironmentEnsure then launches / leaves-in-grace / relaunches-stuck, so a healthy boot is never disturbed and a past-grace flap is re-driven. STOP stays with the toggle / deep-ops until Phase 4; desired=DOWN never yields START/WAIT, so the reconciler cannot fight a legitimate stop, and it will not boot during runroles (holder=INSTALL => desired DOWN). Gated by ServerLifecycleReconciler.ACTUATES (the rollback lever). Hand-off rewired to observe, not poll (SetupProgressActivity): - Removed the three boot latches (moduleRestartKicked / moduleServerUp / moduleServerFailed) and the copy-pasted apiReady re-boot loop (serverUpPoll), plus ensureServerUpForModules. - "up" is now read from the one observed phase (serverObservedUp() over ServerStateRepository); "didn't come back in time" is a timeout on that phase (moduleServerWaitAt). There is no FAILED phase, so a stuck start is STARTING that the reconciler keeps re-driving and the UI shows "taking longer" + Finish - and Finish lands on a Home the reconciler drives live. - onModuleBatchTerminal() sets userWantsOn once (desired=UP); InstallGuard is already cleared and the queue stopped before DONE, so the holder is NONE at that point. Rollback path (ACTUATES=false) boots once here as before. JVM tests: ServerReconcileTest adds the ensuresUp mapping (START/WAIT drive up; NOOP/STOP/ HOLD do not). Verified: :app:testDebugUnitTest + :app:lintDebug BUILD SUCCESSFUL locally. Behavior changes to verify on device: - Reconciler now boots via ANY foregrounded Activity when desired=UP and the box is down (generalizes keep-up; fixes flap on Home). Safe: single idempotent actuator, desired-gated. - A module batch now sets WatchdogEnable (the persisted "server on" intent) true. - Redirect "up" detected via the 3s poll observation (was a 2s bespoke poll) - up to 3s later. Device matrix (must include the timeout branch, not only success): - module install + hand-off -> live Home; - module install + induced dash-node flap post-DONE -> auto-recovers, no manual toggle (5336); - module install where services are slow/never answer -> "taking longer" + Finish; reconciler keeps trying; Finish -> Home stays driven; no dead Home; - runroles in flight -> reconciler does NOT boot (holder STOPPED); - ACTUATES=false -> hand-off boots itself, reconciler log-only (rollback). Rollback: set ServerLifecycleReconciler.ACTUATES=false. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… (5336) Phase-2 device verification found the flap flow regressed: actuation turned a self-healing dash-node blip into an unrecoverable KILL_AND_RELAUNCH loop. Two independent defects, both fixed here (ADR-5343a, approved). D1 - escalate on service downtime, not proot age. EnvironmentEnsure.decide keyed KILL_AND_RELAUNCH on envAgeMs >= BOOT_GRACE_MS, so a mature proot whose dash-node blipped escalated on the first tick - before pdsm's ~3s respawn. It now escalates on how long the services have been continuously observed down while the proot stays present. The clock is one derived field (servicesDownSinceMs) in the single ServerLiveness source, threaded across ticks by the poll (ServerLiveness.next), reset on an observation gap and on boot; a stale/absent snapshot reports -1, which decide treats as "wait". This drops envAgeMs from the decision and replaces the fixed BOOT_GRACE_MS with SERVICE_DOWN_GRACE_MS - a reduction, realizing ADR-5343 2.6. D2 - killOrphan reclaims the orphaned HTTP front. The box's nginx daemonises (setsid, reparents to init) and survives a proot kill, keeping :8085 so a relaunched proot's pdsm start cannot rebind (the loop). reapEnvironmentHttpFront() now reclaims it. This is a compensator for a guest-side defect (services should die with the proot); the clean end-state and the two deviations it carries are recorded in ADR-5343a 9, to retire when the guest-side fix lands - not now. D3 (LibraryActivity:422 second boot owner) is recorded in ADR-5343a 3, deferred to Phase 4 (not pulled forward). JVM tests: EnvironmentEnsureTest re-cast onto service downtime (mature-proot flap WAITs; past-grace relaunches; unknown downtime never kills); ServerLivenessTest adds the servicesDownSinceMs reducer (drop=0, accumulation, observation-gap reset, stale reports -1, services-up / proot-gone clear it). Verified: :app:testDebugUnitTest + :app:lintDebug BUILD SUCCESSFUL locally. Device (a026a310, Luis): flow 2 auto-recovers to UP and rebinds :8085; flow 3 (timeout) verified by hand; flows 1/4/5/6 no regression. Review follow-ups (surfaced, not folded in to keep this to the approved delta): EnvironmentProcess.environmentAgeMs/readStartTicks are now dead (D1 dropped envAgeMs); the lastLiveness reset in doLaunchEnvironment has a second writer (the comment's "single writer" is inaccurate) with a low-probability lost-reset race. Rollback: ServerLifecycleReconciler.ACTUATES=false (device-proven self-healing). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…haned by D1 ADR-5343a D1 replaced the proot-age escalation grace with a service-downtime clock, dropping envAgeMs from EnvironmentEnsure.decide. That left EnvironmentProcess. environmentAgeMs() (public, no remaining callers) and its only user readStartTicks() (private) as dead code. Remove both - the reduction D1 implies. No behavior change; killOrphan / reapEnvironmentHttpFront / isRunning are untouched. Verified: :app:compileDebugJavaWithJavac + :app:lintDebug BUILD SUCCESSFUL. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…eness (Phase 2) lastLiveness has two writers, not one (the review's finding, correcting the stale 'single writer' comment): the poll advances it, and doLaunchEnvironment resets it to null so a fresh proot gets its full grace instead of the old one's inherited service downtime. Before this, the poll read prev (outside any lock), then wrote next(prev,..) ~2.5s later after apiReady() - so a reset that landed during that window was clobbered by the poll's stale prev, and the next ensure-up tick could KILL_AND_RELAUNCH the freshly-booting proot (the 5336 loop, in miniature). Serialise the two writers with livenessLock: probe OUTSIDE the lock (apiReady may block ~2.5s and must not hold it), then read prev and write the next snapshot atomically INSIDE it; doLaunchEnvironment's reset takes the same lock. A reset can no longer be clobbered by a poll that read prev before it. No lock is ever held across the network probe, and the reset's critical section is a single field write, so the UI thread never stalls. The ensure-up decision keeps its lock-free volatile read. Verified: :app:testDebugUnitTest + :app:lintDebug BUILD SUCCESSFUL locally. Device-only (touches the reset path): re-run Flow 2 (kill dash-node on a mature proot) - auto-recovers to UP, rebinds :8085, no relaunch loop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…recovery (ADR-5343a §10) A content service wedged after an environment relaunch loses proot's syscall emulation (epoll_wait -> ENOSYS) and stops serving (kiwix "Unavailable"). Recover it in place, inside the one living proot, via `pdsm restart <svc>` through the dashboard REST core -- never a host-side reap or a proot relaunch (ADR-5343a §10). - services.ts: exact-match allowlist mirroring upstream pdsm_installed_services (dash-node excluded) + setsid-detached restartService actuator. - routes.ts: POST /system/service/:svc/restart (loopback-only, like all /k2go-api). - service-heal.ts: in-proot watcher that HEAD-probes the content tree and heals a present-but-wedged service (404 = not installed -> left alone), cooldown-bounded. - Pure allowlist/cooldown/probe-classification unit-tested; the wedge -> auto-restart recovery is device-only (verified by hand). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…start endpoint REST-facing change (POST /system/service/:svc/restart) — the CHANGELOG rule bumps the version so the app's update-check surfaces it. Missed in 981b214; corrected forward. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ck boundary, absent-vs-down, kiwix-only watcher, service source-of-truth) Align §10 with the implemented dashboard recovery (981b214): - auto-heal is server-side in dash-node; the restart endpoint is loopback-only, so captive-portal clients reflect status but cannot trigger a restart; manual Retry is the on-box Android card (ADFA-4842). - probe classifies absent (404) vs down (5xx/timeout) so an uninstalled service is never restart-looped. - the watcher wires kiwix only today; others added to WATCHED per device-verify. - supported-service list source of truth = iiab/iiab proot_services (pdsm_installed_services); dash-node is the k2go-side exception.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
First stack of the server-lifecycle redesign (see ADR-5343 and ADR-5343a in
controller/docs/). Introduces the process-scoped reconciler as the single owner of"the server should be up," the flap auto-recovery it enables, and in-proot recovery for
content services. Three cohesive parts:
Reconciler foundation (Phases 0–2). One liveness source (
ServerLiveness:/proc+/k2go-api, freshness-windowed) replaces the scattered cached-alive/apiReady//homereads. Adesiredstate derived from facts that already have owners(
installed && healthy && userWantsOn && holder != STOPPED), keyed on theEnvironmentLockholder's execution class (LIVE/STOPPED, reusing ADR-5061'sOperation.ExecutionClass). An app-scopedServerLifecycleReconcilerobserves, thenactuates the module hand-off through
desired(fixes the stuck hand-off, ADFA-5336);Activities become observers.
Flap auto-recovery (D1 + D2, ADR-5343a). The relaunch grace is measured from
service downtime, not proot age, so a transient dash-node death lets pdsm self-heal
(~3 s) instead of being preempted into a
KILL_AND_RELAUNCHloop;killOrphanreclaimsthe orphaned front so a legitimate relaunch can rebind
:8085. Also removes the deadproot-age readers and closes the two-writer race on
lastLiveness.In-proot content recovery (dashboard). A content service wedged after a relaunch
(Kiwix "Unavailable") is recovered in place via
pdsm restart <svc>through thedash-node REST core — a loopback-only endpoint plus a server-side auto-heal watcher —
instead of relaunching proots (which orphans services off proot →
epoll_waitENOSYS).kiwixwired first; others added per device-verify.Why
The server-lifecycle logic had accreted overlapping states and multiple sources of truth
for the lack of a single process-scoped owner. This lands that owner and collapses the
first cluster of the debt; the reasoning and evidence are in ADR-5343 / ADR-5343a.
Verification
Device-verified on OnePlus 7T: the module hand-off, flap auto-recovery (5336) including
the orphan-reclaim, and the kiwix auto-heal (clean-kill, loopback-only security, and the
real orphan-reclaim). Unit tests cover the pure decisions (
ServerLivenessphase table,ServerReconciledesired/intent).:app:testDebugUnitTest+:app:lintDebuggreen;dashboard
tsc+ tests green.Follow-ups (recorded in ADR-5343a)
desired) is the stacked PR on top of this.sleep 3that blocks slow devices — the dashboard/rebuild follow-up.