Restarting an actor that uses a pool leaks broadcast receivers/subscriptions, starving the data pipeline

[Marenz]

Summary

When a long-lived consumer (e.g. a dispatch-driven FcrActor / PowerSettingActor) is stopped and recreated against the same microgrid data pipeline, the receivers and subscriptions it created (grid frequency, BatteryPool power/SoC/power_status, etc.) are not released. Old receivers keep getting messages enqueued but are no longer consumed, so their broadcast queues fill up and the SDK enters a self-sustaining degraded state:

WARNING frequenz.channels._broadcast  Broadcast receiver [...] is full. Oldest message was dropped.
WARNING frequenz.sdk.timeseries._resampling._resampler  No relevant samples found for: ...
WARNING ..._battery_status_tracker  Inverter <id> stopped sending data, last timestamp: ...
INFO    ..._battery_status_tracker  battery <id> changed status ComponentStatusEnum.NOT_WORKING

The components flap to NOT_WORKING and power output drops to 0 even though the underlying microgrid API (nitrogend) keeps streaming telemetry and accepting commands normally. The condition lasts until the whole pipeline is reset.

This was observed in production on an edge controller running frequenz-sdk 1.0.0rc2209, frequenz-channels 1.12.0, frequenz-actor-fcr 0.17.3, frequenz-dispatch 1.0.3, frequenz-app-edge 4.0.0. Bursts last 15–28 minutes and recur at dispatch boundaries where FCR and SET_POWER actors stop/start simultaneously. The first symptom is always a BatteryPool instance was already created for ... warning at the restart.

Root cause

ComponentPool.stop() is an empty stub that releases nothing:

https://github.com/frequenz-floss/frequenz-sdk-python/blob/v1.x.x/src/frequenz/sdk/timeseries/component_pool/_component_pool.py (the stop() method body is only a comment: "This method will do until we have a mechanism to track the resources created through it.")

A ComponentPool (e.g. BatteryPool) hands out:

• receivers created from the shared ChannelRegistry channels (power_status via get_or_create(...), grid frequency via GridFrequency.new_receiver(), formula receivers, SendOnUpdate aggregator receivers for soc/capacity/temperature),
• a power_bounds_subs[channel_name] subscription task per power_status call.

None of these are tracked per pool instance, so when the owning actor is cancelled and a new pool instance is created on restart, the old receivers/tasks are never closed. Broadcast stores receivers as weakrefs, so they would be evicted if collected — but they stay strongly referenced by the still-running shared tasks (power_bounds_subs, FormulaPool/SendOnUpdate engines) and the previous select() set, so they persist and keep filling.

The BatteryPool instance was already created warning at restart is the tell: the new actor re-acquires the shared BatteryPoolReferenceStore (cached forever in _data_pipeline._battery_pool_reference_stores / _known_pool_keys, never released) and stacks new receivers on top of the orphaned old ones.

Impact

• Recurring multi-minute outages (FCR/SET_POWER deliver 0 W) for any deployment that restarts pool-using actors at runtime — most notably dispatch-driven actors.
• Grows worse over repeated restarts (each restart adds another orphaned receiver per shared channel).

Expected

Stopping a ComponentPool (and/or the owning actor) should release the receivers, subscription tasks, and registry channels that instance created, so a restart does not accumulate orphaned consumers. The shared *ReferenceStore should be reference-counted and torn down when the last pool referencing it is stopped.

Proposed fix

1. Track per-instance resources handed out by ComponentPool (receivers, power_bounds_subs task keys, registry channel keys) and release them in a real ComponentPool.stop().
2. Reference-count *ReferenceStore instances in _data_pipeline and release the store (and its _active_methods, FormulaPool, bounds tracker, registry channels) when the last referencing pool stops.
3. Ensure dispatch-driven actor teardown calls the pool's stop().

Workaround

Restart the edge app / pipeline to clear the orphaned receivers (only temporary; the leak re-accumulates on the next actor restart).