Feature Description (功能描述)
Two gaps make it impossible to write an honest barrier for rolling Store replacement (the OnDelete procedure a production image roll needs):
StoreNodeService.register() persists StoreState.Up before the Store runs restoreLocalPartitionEngine(), and a failure there is only logged, so Up never means restored.- A stopped Store stays
Up and listed in all its shard groups until its keep-alive entry expires (store.keepAlive-timeout, 300 s) plus the 60 s patrol. Measured 2026-09-22 on master 83ef9f3 (3+3+3, kind): a Store pod deleted and back after 150 s never left Up; a barrier polling /v1/shardGroups on the PD leader every 2 s reported full membership and one leader per group on every sample of the whole outage.
The consequence: an operator (or any automation) deleting Stores one at a time and gating on PD's view can pass the gate while the previous Store is still down, ending at one live replica per group. Today the only discriminating signal is the Store's own GET :8520/v1/partition/{id} per group, which is ad hoc.
Ask: a Store-side readiness signal that means "registered, partitions restored, raft groups caught up" (the Store counterpart of #3212 for the Server), exposed both as an endpoint suitable for a Kubernetes readinessProbe and to PD; and/or PD exposing heartbeat age (or a suspect state) so a stale Up is distinguishable from a live one. Complements #3136, which asks for the recovery progress metrics; this is the boolean gate on top.
Feature Description (功能描述)
Two gaps make it impossible to write an honest barrier for rolling Store replacement (the OnDelete procedure a production image roll needs):
StoreNodeService.register()persistsStoreState.Upbefore the Store runsrestoreLocalPartitionEngine(), and a failure there is only logged, so Up never means restored.Upand listed in all its shard groups until its keep-alive entry expires (store.keepAlive-timeout, 300 s) plus the 60 s patrol. Measured 2026-09-22 on master83ef9f3(3+3+3, kind): a Store pod deleted and back after 150 s never left Up; a barrier polling/v1/shardGroupson the PD leader every 2 s reported full membership and one leader per group on every sample of the whole outage.The consequence: an operator (or any automation) deleting Stores one at a time and gating on PD's view can pass the gate while the previous Store is still down, ending at one live replica per group. Today the only discriminating signal is the Store's own
GET :8520/v1/partition/{id}per group, which is ad hoc.Ask: a Store-side readiness signal that means "registered, partitions restored, raft groups caught up" (the Store counterpart of #3212 for the Server), exposed both as an endpoint suitable for a Kubernetes readinessProbe and to PD; and/or PD exposing heartbeat age (or a suspect state) so a stale Up is distinguishable from a live one. Complements #3136, which asks for the recovery progress metrics; this is the boolean gate on top.