Observed on a SimulatePubnetMixedLoad run (stellar-core 27.1.1 133f5bcf0, protocol 27). Pod ssc-0942z-2aa52f-sts-node-139-0 ended the run stuck in catchup, so the mission's final EnsureAllNodesInSync fullCoreSet failed.
What the node's log shows, in order:
- It's a non-tier1 leaf with a single peer for the whole run —
TARGET_PEER_CONNECTIONS: 1, PREFERRED_PEERS_ONLY. It spends the first ~2.5 min unable to resolve its preferred peer and rejecting all others, then authenticates one connection:
Unable to resolve peer 'ssc-...-sts-satoshipay-2...': Host not found (authoritative)
Non preferred inbound authenticated peer 172.22.177.113:11625 rejected because all available slots are taken.
Authenticated to 172.22.177.113:11625
- Under load (~1150 tx/ledger, network delay enabled) tx-set fetches from that one peer degrade until it drops out of sync. Apply time stays at 100–400 ms/ledger throughout, so it's starved for data, not slow to apply:
'fetch-e65310' took 13.515000 s
Ledger took 25.794006148 seconds (ledger 116)
'fetch-3cd058' took 40.518002 s
Herder WARNING Lost track of consensus
Lost sync, local LCL is 123, network closed ledger 126
- Catchup then can't proceed.
FullPubnetCoreSets sets historyNodes = Some([]) (StellarNetworkData.fs:580), so the only archive is the node's own local at /data/history. It last published checkpoint 63 before losing sync, so the checkpoint it now needs was never written, and it retries indefinitely:
Catching up to ledger 127: Downloading state file history/00/00/00/history-0000007f.json
cp: cannot stat '/data/history/history/00/00/00/history-0000007f.json': No such file or directory
Missing HAS for ledger 127: maybe stale archive local
That message repeats with growing backoff for the remaining ~11 minutes of the run. Meanwhile it keeps buffering externalized ledgers (mSyncingLedgers reaches 66) without ever applying them, and never returns to Synced!.
So in these missions a single loss of sync is terminal for that node, and the mission fails at the end regardless of how the rest of the network behaved.
Observed on a
SimulatePubnetMixedLoadrun (stellar-core 27.1.1133f5bcf0, protocol 27). Podssc-0942z-2aa52f-sts-node-139-0ended the run stuck in catchup, so the mission's finalEnsureAllNodesInSync fullCoreSetfailed.What the node's log shows, in order:
TARGET_PEER_CONNECTIONS: 1,PREFERRED_PEERS_ONLY. It spends the first ~2.5 min unable to resolve its preferred peer and rejecting all others, then authenticates one connection:FullPubnetCoreSetssetshistoryNodes = Some([])(StellarNetworkData.fs:580), so the only archive is the node's ownlocalat/data/history. It last published checkpoint 63 before losing sync, so the checkpoint it now needs was never written, and it retries indefinitely:That message repeats with growing backoff for the remaining ~11 minutes of the run. Meanwhile it keeps buffering externalized ledgers (
mSyncingLedgersreaches 66) without ever applying them, and never returns toSynced!.So in these missions a single loss of sync is terminal for that node, and the mission fails at the end regardless of how the rest of the network behaved.