Reconnect dead Cast connections instead of failing silently

[sevensrig]

The bug

Volume control stopped working with no visible symptom — faders moved, /status reported sensible levels, nothing was marked unavailable, and the speakers ignored everything. The journal showed requests arriving and the ratio being computed, but no writes:

=== Volume adjust (ratio) delta=-0.050 ===
[nest_audio] set_volume failed:
[google_home] set_volume failed: [chromecast] set_volume failed:

_init_casts() opens one pychromecast connection per speaker at startup only, and nothing ever reopens them. All three had died, so every write threw.

Two things kept it hidden:

• cast.status keeps serving the last cached levels after the socket dies, so /status looked healthy and unavailable reported nothing.
• Volume writes are asynchronous now, so the response returns ok:true the moment a target is queued. The failure only ever reached the journal. That regression came in with b427c33, which traded visible per-speaker failure for a 14× latency win without giving the failure another route out.

Restarting the service was the only cure, and the only symptom was a fader that moves while nothing happens.

The fix

• Reconnect on failure. Each write gets two attempts; if the first throws, the connection is dropped and reopened, so turning the dial is enough to recover. The disconnect runs on a throwaway thread because tearing down a dead socket can block and the writer must not stall. Reconnects are per-speaker, so one dead speaker can't hold up the others.
• Real liveness. _speaker_unavailable() now checks socket_client.is_connected, so a dropped link is reported immediately instead of waiting for a write to fail. The fader dims and shows -- rather than looking live.
• Surface async failures. The last write error per speaker is tracked and reported through /status, which is now the only channel that can carry it.

Also logs the exception type — a dead pychromecast socket raises with an empty str(), which is why the failures above printed no reason at all.

Testing

iptables isn't installed on the Pi and the kernel lacks INET_DIAG_DESTROY, so neither blocking traffic nor ss -K could kill a live socket. Verified the logic with fakes standing in for connections instead:

[1] dead socket reported before any write : 'offline'
    reconnect attempted                   : ['nest_audio']
    volume landed on new connection       : [0.42]
    write error cleared                   : None
[2] speaker gone -> write error           : 'offline'    (no infinite retry)
    CASTS entry cleared                   : True
[3] other speaker's write error           : None         (failures don't cross speakers)

On the device: connections open at startup and adjusts land as before —

[nest_audio] connected to 10.0.0.161 (cast)
[google_home] connected to 10.0.0.88 (cast)
[chromecast] connected to 10.0.0.47 (cast)
[nest_audio] -> 0.14   [chromecast] -> 0.26   [google_home] -> 0.24

Not covered: the real-world path where a socket dies mid-session is verified by construction rather than by observation, for the tooling reasons above. Recovery is also write-triggered — a speaker that drops while idle shows as dimmed but only reconnects on the next dial turn. A periodic health check would close that gap if it proves annoying.

🤖 Generated with Claude Code