Granular network state events for backup/restore

[puddly]

Adds a few new commands:

• LoadRouteTable([destination: Nwk, next_hop: Nwk, path_cost: u8]): load a batch of routes
• LoadSourceRoutes([destination: Nwk, relays: [Nwk]]): load a batch of source route relays.
• RouteChanged(destination: Nwk, next_hop: Nwk, path_cost: u8, removed: bool): a route to destination has changed or has been removed.
• RouteRecord(destination: Nwk, relays: [Nwk]): a node sent a route record notification after forwarding some data.
• ApsFrameCounter(frame_counter: u32): APS frame counter has incremented by 100

This allows zigpy to fully maintain a clone of the network state at runtime without performing periodic backups. On my network it's only a few events per minute. This allows zigpy to re-seed the entire Ziggurat state on startup.

Implemented in zigpy/zigpy-ziggurat#3 + zigpy/zigpy#1856

[burmistrzak]

We may not be able to get XNCP support merged anytime soon but at least the new ziggurat adapter for Z2M will technically be able to restore routes. 😏

AFAIK,zigbee-herdsman entirely lacks the ability to restore or backup route tables. So we'll have to work on that too... 😬

[puddly]

I'm debating whether or not it would be better to break the "no state" seal and maybe persist some of this info to disk as cache, to simplify implementations? Right now this hardcodes "one NWK one route" but in the future this will not be the case...

On the other hand, the whole purpose of this change is to bootstrap the stack after reloading, as otherwise things are slow for a few minutes. We could also just...not reset Ziggurat after connecting, to sidestep this issue entirely. Or accept "incomplete" cache restoration as an acceptable limitation.

[burmistrzak]

I'm debating whether or not it would be better to break the "no state" seal and maybe persist some of this info to disk as cache, to simplify implementations? Right now this hardcodes "one NWK one route" but in the future this will not be the case...

Being able to backup the entire network incl. routes is pretty huge, IMHO.
However, I'm also a bit conflicted about having the adapter/driver handle all that because explicit driver and app support would again be required to make this work reliably...

On the other hand, the whole purpose of this change is to bootstrap the stack after reloading, as otherwise things are slow for a few minutes. We could also just...not reset Ziggurat after connecting, to sidestep this issue entirely. Or accept "incomplete" cache restoration as an acceptable limitation.

I like the idea of ziggurat being stateless by default.
But being totally honest here, it will be difficult for me alone to make all the necessary changes to ZH plumbing to handle route table management in time for a v1.0.0 release. 😳

[puddly]

However, I'm also a bit conflicted about having the adapter/driver handle all that because explicit driver and app support would again be required to make this work reliably...

Yeah. At the end of the day this is all just cache, the network won't really suffer if this data isn't persisted. I think the actual impact of not storing/restoring the route cache is minimal, IMO: EmberZNet already keeps it all in RAM and discards it when you reset the adapter.

zigpy and I think herdsman reset adapters pretty religiously on startup. With Rust's memory safety, maybe we can skip a startup reset if the current network settings are already correct, since all expected state is effectively contained to a few tables? That way, a host-side app can keep running indefinitely and survive Zigbee integration restarts. The only "downtime" would be after a full reboot, in which case you end up with a slow start scenario like you have with EmberZNet (though I imagine we can maybe do some hacks to speed up a cold start or at least make one feel more responsive).

[burmistrzak]

zigpy and I think herdsman reset adapters pretty religiously on startup. With Rust's memory safety, maybe we can skip a startup reset if the current network settings are already correct, since all expected state is effectively contained to a few tables?

Both libraries were designed with finicky NCPs in mind, so it makes sense to at least offer the (default?) option to skip reset for Zigbee-on-Host setups.

That way, a host-side app can keep running indefinitely and survive Zigbee integration restarts. The only "downtime" would be after a full reboot, in which case you end up with a slow start scenario like you have with EmberZNet (though I imagine we can maybe do some hacks to speed up a cold start or at least make one feel more responsive).

Good compromise, IMO.

It's unlikely that network parameters change during Z2M/ZHA reset, but ziggurat could probably detect such a scenario and discard its cache automatically?

[puddly]

Since these are strictly new commands, I'm going to merge this PR. We can remove them in a v1.0.0 release if they don't find much use or pose issues.