fix(checkpoint): harden generation-prefix recovery

[macandro96]

Summary

This PR hardens the generation-prefix recovery mechanism introduced by #4319 and adds recovery-parity coverage for the races that occur around checkpoint publication, restart, and terminal completion.

The main invariant is:

A restored TQ prefix remains protected while its replacement model call can still consume it. It becomes eligible for cleanup only after a canonical terminal replacement has been sealed or the recovery ledger has definitively moved past that attempt.

What this fixes

• Protects restored generation-cut rows from successor-snapshot garbage collection while a replacement call is still using them.
• Maps generation_prefix_cut storage references to their owning recovered continuation instead of treating them as ordinary committed-turn rows.
• Preserves the correct cleanup order: stage the canonical terminal row first, then retire the obsolete prefix chunks.
• Adds separate token-count and admission-digest mismatch diagnostics, including safe non-token metadata, so invalid restored prefixes fail closed and are debuggable without logging token contents.
• Tightens coordinator drain readiness and retry settings used by recovery.
• Stabilizes structured-output and repeated-restart parity checks.

Successor-checkpoint lifecycle

flowchart LR
    A[Checkpoint A contains prefix P] --> B[Restore A]
    B --> C[Replacement call consumes P]
    C --> D{Replacement sealed?}
    D -- No --> E[Keep P protected in TQ]
    E --> C
    D -- Yes --> F[Canonical terminal row T is durable]
    F --> G[Allow P cleanup]
    G --> H[Checkpoint B references T, not P]

Loading

Recovery-parity coverage

• Consecutive crash and restart cycles.
• Completion during or immediately after a prefix cut.
• Successor checkpoints taken while a restored model call is in flight.
• Already-terminal and exhausted restored prefixes.
• Finish/stop-reason and reasoning-token accounting.
• Structured-prefix restart/fallback behavior.
• Coordinator drain readiness and temporary/incomplete checkpoint handling.

Scope

• This PR intentionally does not add sharded Gym or multi-replica ownership; those remain downstream integrations.
• It does not change the storage architecture introduced by feat(checkpoint): add generation-prefix recovery #4319. TQ remains authoritative for token payloads and Gym remains authoritative for lineage and recovery coordinates.

Stack

• Depends on feat(checkpoint): add generation-prefix recovery #4319.
• Followed by checkpoint control-plane optimizations and sharded Gym integration.

Validation

• git diff --check passed.
• Python compilation passed for the checkpoint orchestration modules.
• Parity and end-to-end recovery suites still need to run on the cluster against the published stack.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.