Summary
On restart, the daemon has a window where nothing is listening on its port, so incoming connections are dropped (refused at the kernel / 502'd by the front proxy) with no corresponding entry in redbot's own log — the connections never reach the app to be logged. Observed as connections being dropped "for a while" during a deploy, then stabilizing once startup finished.
Mechanism
redbot.service is Type=simple with Restart=always and no socket-activation unit, so on restart systemd fully stops the old process before starting the new one — there is no listen-socket handoff. Two serial delays stack up inside that no-listener window:
-
Old-process graceful drain. SIGTERM → graceful_shutdown() (redbot/daemon.py:219) stops accepting and waits for in-flight exchanges to finish. Those include upstream HTTP tests that can take several seconds, so the drain is not instant, and systemd won't start the replacement until the old process exits.
-
New-process init before it binds. The listen socket is not opened until thor.http.HttpServer(...) is constructed at the very end of RedBotServer.__init__ (redbot/daemon.py:123). Everything before it runs with no socket up — notably load_signer, resource_files, and walk_files(extra_base_dir) (redbot/daemon.py:100), which walks and reads every extra file into memory synchronously. Under the unit's CPUQuota=60% and MemoryHigh=50M/MemoryMax=60M, a cold process doing that import + file-read + regex work is throttled, stretching the pre-bind phase.
Total downtime = drain + init, all with the port dark. This is why nothing appears in redbot's log: it can only log connections it accepts, and there was no listener. (Contrast the earlier watchdog SIGABRT, which died loudly with a stack dump — that was the process present and failing; this is the process simply not there yet.)
Where to confirm
- Reverse-proxy access/error log (the earlier crash dump showed a
localhost:8000 connection, i.e. a TLS-terminating proxy in front) — the 502/504s land here with timestamps.
journalctl -u redbot — measure the stop→start gap; redbot's own log is correctly empty across the interval.
Proposed fixes (cheapest → most correct)
-
Early bind. Move the HttpServer construction to near the top of __init__, before walk_files/load_signer. Binding early lets the kernel queue incoming connections in the listen backlog during the rest of init instead of refusing them; the handler only runs after thor.run(), and everything it reads (static_root, extra_files, …) is still set by then. Converts "refused" into "slightly delayed." Small, low-risk.
-
Socket activation. Add a redbot.socket unit so systemd owns the listen fd and holds connections across restarts. Bigger lift: thor would need a path to adopt an inherited fd (sd_listen_fds / LISTEN_FDS), which it does not appear to have today.
Filed by Claude (Claude Code) at the maintainer's request. Root cause was diagnosed collaboratively in a session reviewing a deploy where startup dropped connections; the maintainer reviewed and directed the analysis. No code has been changed.
Summary
On restart, the daemon has a window where nothing is listening on its port, so incoming connections are dropped (refused at the kernel / 502'd by the front proxy) with no corresponding entry in redbot's own log — the connections never reach the app to be logged. Observed as connections being dropped "for a while" during a deploy, then stabilizing once startup finished.
Mechanism
redbot.serviceisType=simplewithRestart=alwaysand no socket-activation unit, so on restart systemd fully stops the old process before starting the new one — there is no listen-socket handoff. Two serial delays stack up inside that no-listener window:Old-process graceful drain. SIGTERM →
graceful_shutdown()(redbot/daemon.py:219) stops accepting and waits for in-flight exchanges to finish. Those include upstream HTTP tests that can take several seconds, so the drain is not instant, and systemd won't start the replacement until the old process exits.New-process init before it binds. The listen socket is not opened until
thor.http.HttpServer(...)is constructed at the very end ofRedBotServer.__init__(redbot/daemon.py:123). Everything before it runs with no socket up — notablyload_signer,resource_files, andwalk_files(extra_base_dir)(redbot/daemon.py:100), which walks and reads every extra file into memory synchronously. Under the unit'sCPUQuota=60%andMemoryHigh=50M/MemoryMax=60M, a cold process doing that import + file-read + regex work is throttled, stretching the pre-bind phase.Total downtime = drain + init, all with the port dark. This is why nothing appears in redbot's log: it can only log connections it accepts, and there was no listener. (Contrast the earlier watchdog SIGABRT, which died loudly with a stack dump — that was the process present and failing; this is the process simply not there yet.)
Where to confirm
localhost:8000connection, i.e. a TLS-terminating proxy in front) — the 502/504s land here with timestamps.journalctl -u redbot— measure the stop→start gap; redbot's own log is correctly empty across the interval.Proposed fixes (cheapest → most correct)
Early bind. Move the
HttpServerconstruction to near the top of__init__, beforewalk_files/load_signer. Binding early lets the kernel queue incoming connections in the listen backlog during the rest of init instead of refusing them; the handler only runs afterthor.run(), and everything it reads (static_root,extra_files, …) is still set by then. Converts "refused" into "slightly delayed." Small, low-risk.Socket activation. Add a
redbot.socketunit so systemd owns the listen fd and holds connections across restarts. Bigger lift: thor would need a path to adopt an inherited fd (sd_listen_fds/LISTEN_FDS), which it does not appear to have today.Filed by Claude (Claude Code) at the maintainer's request. Root cause was diagnosed collaboratively in a session reviewing a deploy where startup dropped connections; the maintainer reviewed and directed the analysis. No code has been changed.