DO Wall-Time nightly check — triage of the 2026-08-18 failure

[defangdevs]

Timeline of the daily runs (as of filing, 2026-08-18)

• Aug 4 to Aug 14: FAIL. The RESOURCE_PREFIX GitHub Environment variable was empty. The check exited before it read any data. The variable is set now (sf209b3), so this is fixed.
• Aug 15: FAIL. Cron liveness: no cron.completed event in the last 3 hours.
• Aug 16: PASS.
• Aug 17: FAIL. Cron liveness again: no cron.completed event in the last 3 hours.
• Aug 18 (run 32121731222): FAIL. Cron liveness is OK. Wall-time is OK. One Durable Object shows an alarm invocation-rate regression:
  • sf209b3-api-prod / 7671945667c14bc68e00d8edb43b1993
  • recent: 11.75 requests/hour (282 requests in 24 h)
  • baseline: 5.43 requests/hour (913 requests in 7 days) — ratio 2.16x, threshold 2x

My read

The baseline week (Aug 10-17) includes the days when the cron was stalled (the liveness failures above). A stalled cron lowers the baseline alarm rate. When the cron recovered, the recent rate looked like a 2x regression against that low baseline. If this read is correct, the alert clears by itself around Aug 22-24, when the stalled days leave the 7-day baseline window.

Actions needed (no Cloudflare access from this side)

1. Check the Cloudflare dashboard for sf209b3-api-prod: is ~12 alarm invocations/hour the normal healthy rate, or is one object in a retry loop?
2. The cron liveness failures on Aug 15 and Aug 17 look like real production stalls, separate from the rate alert. Worth a look at what stopped cron.completed on those days.

No code change was needed in the repo for the 2026-08-18 failure.

History

Originally tracked at defangdevs/nixos-defang-ca#22 (filed there because Issues were off in this repo). Moved here now that Issues are enabled.

[defangdevs]

2026-09-08 run — much larger spike, different Durable Object

Run: https://github.com/DefangLabs/simple-agent-manager/actions/runs/34210266989

Cron liveness: OK. Wall-time: OK. Invocation-rate check found 2 regressions:

• sf209b3-api-prod / 1b4dca83b9aa45f7b783fed37190fc66
  • ratio: 90.25x
  • recent: 86401.63 requests/hour (2,073,639 requests in 24h)
  • baseline: 957.40 requests/hour (160,843 requests in 7 days)
• sf209b3-api-prod / d0433aa7d4744e00a21b3fd882cc74aa
  • ratio: 5.25x
  • recent: 3.63 requests/hour (87 requests)
  • baseline: 0.69 requests/hour (116 requests)

This does not look like the same "low-baseline artifact" as the 2026-08-18 entry above:

• The ratio (90x) and the raw volume (2.07M requests to one object in 24h, vs 160k over the prior 7 days) are both far larger than the earlier 2.16x case.
• The object ID (1b4dca83b9aa45f7b783fed37190fc66) is different from the one flagged on 2026-08-18 (7671945667c14bc68e00d8edb43b1993).
• The prior week's runs were mostly PASS (2026-09-01, 09-02, 09-06), so the 7-day baseline here is not depressed by stalled-cron days the way the August one was.
• No code has merged to main since PR Main 3 #27 (2026-05-21), so this is not a new deploy causing extra invocations — it looks like a runtime/production change (e.g. one Durable Object stuck retrying, or a real traffic increase to that object).

I don't have Cloudflare dashboard access from this session, so I can't tell whether this is a retry loop (cost/perf risk) or expected traffic. Flagging here since this is the open tracking issue for these nightly failures — could one of you check the CF dashboard for object 1b4dca83b9aa45f7b783fed37190fc66 under sf209b3-api-prod?

[defangdevs]

2026-09-11 run — sustained elevation, not a transient spike; Aug 18 object also recurring

Run: https://github.com/DefangLabs/simple-agent-manager/actions/runs/34584388097

Cron liveness: OK on 09-09, 09-10, 09-11. Wall-time: OK. Invocation-rate check results for the last four days:

Date	1b4dca83...fc66 ratio	recent req/hr	baseline req/hr	7671945667...1993 ratio
09-08	90.25x	86,401.63	957.40	—
09-09	5.54x	73,311.17	13,237.61	—
09-10	3.25x	77,082.04	23,690.07	2.73x (12.21 req/hr)
09-11	2.37x	82,104.04	34,647.29	4.38x (23.71 req/hr)

Two things worth flagging:

1. 1b4dca83b9aa45f7b783fed37190fc66 is not returning to normal — the baseline is catching up to it. Recent-rate has stayed flat at ~73k-86k req/hour for four straight days while the 7-day baseline climbs (957 → 13,238 → 23,690 → 34,647) as the old low-traffic days roll out of the window. The ratio is dropping only because the elevated rate is becoming "the new normal" in the baseline, not because traffic is subsiding. If this is a retry/alarm loop rather than legitimate traffic, it's now been running at ~2M requests/day for 4+ days.
2. The original 2026-08-18 object (7671945667c14bc68e00d8edb43b1993) is back, on both 09-10 and 09-11. The original theory in this issue ("clears by itself around Aug 22-24 once the stalled-cron days leave the baseline window") hasn't held — we're well past that window and it's still firing.

Still no Cloudflare dashboard access from this side to tell whether 1b4dca83b9aa45f7b783fed37190fc66 is a stuck retry loop (cost/perf risk, worth killing) or a real traffic increase (in which case the check's threshold/baseline window needs adjusting so it stops paging every night). Given it's sustained for 4 days now, could one of you take a look at the CF dashboard for that object under sf209b3-api-prod?

[defangdevs]

2026-09-12 run — 1b4dca83...fc66 gone from the report; d0433aa7 now trips both checks

Run: https://github.com/DefangLabs/simple-agent-manager/actions/runs/34685801075

Cron liveness: OK. Two check types fired today, both against d0433aa7d4744e00a21b3fd882cc74aa:

Wall-time regression (new — this object hasn't shown up in wall-time before, only invocation-rate):

• d0433aa7d4744e00a21b3fd882cc74aa / alarm
  • ratio: 3.26x
  • recent avg P99: 1,667ms (max 6,157ms, 31 requests)
  • baseline avg P99: 511ms (max 511ms, 2 requests)

That baseline is 2 requests total over 7 days — same "low-baseline artifact" shape as the original 2026-08-18 entry, so I wouldn't read 3.26x as a real regression on its own.

Invocation-rate regressions (2):

• 7671945667c14bc68e00d8edb43b1993: ratio 2.46x, recent 21.17 req/hr (508 reqs), baseline 8.59 req/hr (1,443 reqs) — this is the original Aug 18 object, still recurring as noted on 09-10/09-11.
• d0433aa7d4744e00a21b3fd882cc74aa: ratio 2.37x, recent 3.63 req/hr (87 reqs), baseline 1.53 req/hr (257 reqs)

Notable change: 1b4dca83b9aa45f7b783fed37190fc66 — the object that spiked to 90.25x on 09-08 and stayed elevated at 73k-86k req/hour through 09-11 — does not appear anywhere in today's output. Either it dropped back under the min-requests threshold, or its recent activity genuinely returned to baseline-normal. If it's the latter, that's good news and suggests the 09-08 spike was transient (a retry storm that resolved on its own) rather than a permanent traffic shift — worth confirming on the CF dashboard if anyone gets a chance, since it bears on whether the 09-08/09-09 alarm should have paged at all.

Still flagging the standing ask from this thread: none of these sessions have Cloudflare dashboard access, so we can't independently tell a genuine retry/alarm loop from legitimate traffic — someone with CF access checking objects d0433aa7d4744e00a21b3fd882cc74aa and 7671945667c14bc68e00d8edb43b1993 under sf209b3-api-prod would let us close this out one way or another.

[defangdevs]

2026-09-13 run — first cron liveness FAILURE since tracking began; 7671945667...1993 recurs again

Run: https://github.com/DefangLabs/simple-agent-manager/actions/runs/34750513234

Two things fired today, and one of them is new:

1. Cron liveness FAILED for the first time in this thread's history.

Cron liveness failure: no cron.completed event observed within 3h.

I checked the logs for every daily run back through 09-06 and all of them reported Cron liveness: cron.completed observed within 3h. — today is the first failure. This is a different signal from the invocation-rate/wall-time noise this issue has otherwise been tracking: it means the Workers observability telemetry query found no cron.completed event for sf209b3-api-prod in the last 3 hours, which is consistent with an actual scheduled (cron-triggered) Worker execution not completing recently, rather than a baseline-window artifact.

2. Invocation-rate regression — same object as 2026-08-18/09-10/09-12:

• sf209b3-api-prod / 7671945667c14bc68e00d8edb43b1993
  • ratio: 2.49x
  • recent: 21.08 req/hr (506 requests)
  • baseline: 8.46 req/hr (1,421 requests)

This matches the recurring low-magnitude pattern already noted for this object on 09-10 (2.73x) and 09-12 (2.46x) — no new theory needed there.

As before, none of these triage sessions have Cloudflare dashboard/Workers observability access, so I can't independently confirm whether the cron liveness failure reflects a real stopped/stuck cron trigger in production or a telemetry-query gap. Given this is the first liveness failure in this issue's history (as opposed to the recurring invocation-rate noise), it seems worth someone with CF access checking the sf209b3-api-prod Worker's cron trigger history directly — if the production cron has actually stopped completing, that's a live incident, not a threshold-tuning question like the rest of this thread.

[defangdevs]

2026-09-14 run — cron liveness FAILED again, second day in a row; wall-time/invocation-rate both clean

Run: https://github.com/DefangLabs/simple-agent-manager/actions/runs/34828764892

Recent series: 14 | Baseline series: 137 | Skipped series: 8
No Durable Object wall-time regressions detected at 2x threshold.
Recent invocation-rate series: 4 | Baseline invocation-rate series: 4
No Durable Object invocation-rate regressions detected at 2x threshold.
Cron liveness failure: no cron.completed event observed within 3h.

So today is clean on both of the noisy signals this thread has mostly been tracking (wall-time, invocation-rate) — the only failure is cron liveness, same as 09-13. That makes it two consecutive days with no cron.completed event inside the 3h window, which reads less like a one-off telemetry gap and more like the scheduled Worker actually stopped completing (or stopped running) sometime around 09-13.

Still no Cloudflare dashboard/Workers observability access from this side, so I can't confirm directly — but given it's now sustained for 2 days, this seems worth treating as a live production incident rather than more threshold-tuning noise: could someone with CF access check the sf209b3-api-prod Worker's cron trigger execution history to see when it last actually ran?