A battery disconnect resets the RTC, and the stack writes backdated samples that nothing notices

[Gerrrt]

Replacing prometheus's laptop cell on 2026-09-18
(#454) disconnected the battery
that backs the RTC. The machine came back with no idea what time it was, wrote
about fourteen minutes of Prometheus samples at the wrong timestamps, and
nothing in the estate noticed — including the rule written for clock
problems.

What happened, from the host's own journal and TSDB

Real time	What the machine believed	What was recorded
14:53:37	14:53:37	clean shutdown, last good samples
~17:58	2026-07-28 15:04:45	post-swap boot; journal boot -1 opens here
~17:58	stepped to 14:53:37	systemd-timesyncd: "System clock time unset or jumped backwards, restored from recorded timestamp"
~17:58–18:12	14:54:30–15:08	~14 min of real operation, written backdated
18:12	18:12	NTP syncs, clock steps forward ~3 h 4 m

systemd-timesyncd restores the clock from its own clock file, whose timestamp
is the last time it ran — the second the machine went down. So the post-swap
boot is not merely wrong, it is wrong in a way that lands inside the outage it
is meant to be after.

Why this matters more than a cosmetic timestamp

1. The TSDB shows a 184-minute hole at 15:08–18:12 that is not downtime. The
   true window was 14:53:37 to about 18:12, roughly 3 h 18 m. Neither figure is
   recoverable from the series: one is an artefact of the step, the other was
   never recorded at all.
2. Real post-swap operation is filed inside the outage. HostOnBattery
   genuinely fired for prometheus — the machine came back unplugged — and it is
   stamped 14:56 pending, 14:57 firing, which is three hours before it happened
   and inside a window the series otherwise reads as dead.
3. replace-the-laptop-cell.md's step 6 tells the operator to measure the
   outage with query_range. After a battery swap that instruction returns a
   confident wrong answer, which is the failure mode this repository keeps
   warning about one level up. That runbook is being corrected to say so; this
   issue is about whether anything should detect it.
4. oracle is next in line for the same procedure and will do the same
   thing.

HostClockSkew is blind to this, and the reason is structural

host.rules.yaml:74 —

- alert: HostClockSkew
  expr: abs(node_timex_offset_seconds) > 0.5
  for: 10m

node_timex_offset_seconds peaked at 0.004 s across the whole event, and
HostClockSkew has never fired in the 30-day retention. That is not bad luck.
The metric is the kernel PLL's current correction estimate; once timesyncd has
restored a clock that is stable and three hours wrong, there is nothing to
correct and the offset is tiny. The rule detects a clock that is drifting, not
one that is wrong.

The signal that was there all along is one field over:

node_timex_sync_status{instance="prometheus"}

It went to 0 at 14:55 and back to 1 at 18:12 — precisely the backdated
window. Nothing in stacks/observability/prometheus/rules/ reads it.

The question

Should something notice, and what should it read? Candidates, none of them yet
argued for:

• node_timex_sync_status == 0 for some for: — exactly what went unread here,
  and it would also catch an NTP outage that has not yet produced drift. It will
  fire briefly on every ordinary boot, so the for: is the whole design.
• A backward jump in node_boot_time_seconds, which would catch this case
  specifically but is a narrower claim.
• Nothing, on the grounds that a battery swap is a supervised operation and the
  runbook now says to read the journal. That is a legitimate answer and should
  be written down if it is the one chosen.

There is a real cost to (1) and (2): the monitoring host writing its own
timestamps means Prometheus may be down for the interesting part, which is what
happened here. Whatever is chosen should say what it can and cannot see.

Not in scope of the change that raised this

No rule lands with the #454 fit write-up. That change corrects
replace-the-laptop-cell.md so the operator takes the window from
journalctl --list-boots rather than query_range, and points here.

🤖 Generated with Claude Code

[Gerrrt]

One datum for the "will oracle do the same thing" line, from planning #531 (PR #540), read on 2026-09-19 rather than assumed:

• oracle's kernel RTC is rtc_cmos, and the Dell Inspiron 15-3565 has a separate CR2032 coin cell on the board. Dell's service manual and the parts-people teardown put it behind the main pack, the optical drive, the keyboard and the palmrest — so a swap of the latched main pack never disconnects it. Dell's own note is that removing the coin cell resets BIOS setup; the main pack is not in that sentence.
• The MacBook's RTC is backed by the main cell, which is why the 2026-09-18 swap did what the body above describes. oracle is therefore expected not to reproduce it.
• journalctl -u systemd-timesyncd on oracle has no "jumped backwards / restored from recorded timestamp" line across its last five boots, and timedatectl shows the RTC in step with the system clock.

replace-the-laptop-cell.md now tells the operator to check journalctl --list-boots and the timesyncd journal for the post-swap boot on oracle and to record the result here either way. A pass there is a second data point for whichever of the three candidates this issue picks; it does not pick one.

🤖 Generated with Claude Code

[Gerrrt]

Answered: something should notice, and it reads node_timex_sync_status.

HostClockUnsynchronised landed in #548 — node_timex_sync_status == 0 for 5m, warning, category correctness, routed like HostClockSkew — and was deployed by hand from the main checkout on 2026-09-20. Read back from the live Prometheus after converge:

HostClockUnsynchronised ok inactive

Why the offset could not see it. node_timex_offset_seconds read exactly 0 for the whole backdated window: a clock that is wrong but stable has nothing to correct. node_timex_sync_status was 0 from the first post-swap scrape (14:54:27 TSDB time) to the NTP step (15:08:12), 13.8 minutes. This is ADR-0029's pair on Linux: one rule reads the offset, the other reads whether there is a source at all.

Why for: 5m, measured rather than picked. Over the 30 days of retention on 2026-09-19 the estate booted eight times — Saruman three, oracle three, prometheus twice — and, zoomed to 15s resolution around each boot, produced not one sample of 0: timesyncd syncs before the first 60s scrape on an ordinary boot. The only run of 0 in retention was this event. So the premise above that it would fire briefly on every boot did not hold. Five minutes is five scrapes above the measured normal, would have fired about six minutes into the window and roughly eight minutes before NTP repaired it; 15m would have missed it.

The other candidates. A backward jump in node_boot_time_seconds was rejected: the series never jumped backward — the 2026-07-28 value never reached a scrape, the restored 14:53:27 is later than the previous boot, and the NTP step moved it forward to 18:00:52. HostRebooted did fire, at info, into null. "Nothing" was rejected because oracle is next for the same swap and because the sync flag also covers an NTP path that dies quietly (the kernel re-raises STA_UNSYNC about nine hours after the last update).

What it can and cannot see, written into the rule comment and replace-the-laptop-cell.md: for a remote-writing host other than the monitoring host it fires in real time and sees the whole window. For prometheus itself it sees only the slice in which Prometheus ran on the wrong clock; the notification is real-time, the alert's own record carries the wrong clock, and the hours the stack was down are invisible to it. The runbook instruction stands: take the window from journalctl --list-boots, not query_range.

Tests: an event-shaped fixture plus a quiet case; both ledger mutations (== 0 -> == 1, for: 5m -> 0m) fail a test. No ADR — the principle is ADR-0029's and the rule's comment carries the argument.

🤖 Generated with Claude Code