{186552229} Keep the sc_list when the master downgrades mid-DDL (fixing timepart_retro)

[WalidNejmi]

Summary

Fix a race where a master downgrade during a multi-table schema change could delete the persisted sc_list before the new master had a chance to resume it.

This showed up as a timepart_retro flake: the old master aborted the DDL, removed the resume state, and the new master found nothing to continue, so the retroactive partition shards were never created.

The fix clears OSQL_FLAGS_SCDONE on the ERR_NOMASTER path so the old master's abort callback preserves the sc_list. The new master can then resume the DDL and clean up the list normally when the operation finishes.

Background (if unfamiliar)

timepart_retro converts an existing table into a retroactive time partition using the multi-table DDL path.

For multi-table DDLs, Comdb2 persists an sc_list in llmeta before starting the schema changes. If the master changes while the DDL is running, the new master uses this list to resume the operation.

flowchart LR
    A[Old master starts multi-table DDL] --> B[Persist sc_list]
    B --> C[Schema change runs]
    C --> D[Master downgrades]
    D --> E[New master reads sc_list]
    E --> F[Resume DDL]

Loading

Bug

When a schema change detected a master downgrade inside bplog_schemachange_wait(), it returned ERR_NOMASTER and cleared sc_pending, intending to leave the persistent schema-change state for the new master.

However, OSQL_FLAGS_SCDONE remained set.

The transaction then aborted and osql_scdone_abort_callback() deleted the persisted sc_list. If that delete happened before the new master resumed the DDL, there was no state left to resume.

flowchart TD
    A[Master downgrade during DDL] --> B[Return ERR_NOMASTER]
    B --> C[sc_pending cleared]
    C --> D[SCDONE still set]
    D --> E[Abort callback runs]
    E --> F[Delete sc_list]
    F --> G[New master finds no DDL to resume]
    G --> H[timepart_retro fails]

Loading

This caused the flaky timepart_retro failure where the retroactive partition shards were never created.

Fix

Clear OSQL_FLAGS_SCDONE when bplog_schemachange_wait() returns ERR_NOMASTER:

if (rc == ERR_NOMASTER) {
    iq->sc_pending = NULL;
    iq->osql_flags &= ~OSQL_FLAGS_SCDONE;
}

This prevents the old master's abort callback from deleting the sc_list.

The new master can then resume the DDL normally. Once the resumed operation finishes or fails, the normal resume path sets SCDONE again and performs the usual cleanup.

flowchart TD
    A[Master downgrade] --> B[Return ERR_NOMASTER]
    B --> C[Clear sc_pending]
    C --> D[Clear SCDONE]
    D --> E[Abort callback preserves sc_list]
    E --> F[New master reads sc_list]
    F --> G[Resume DDL]
    G --> H[Normal commit or abort cleanup]

Loading

This also matches the existing handling of SC_MASTER_DOWNGRADE in the schema-change start paths.

Testing

Before the fix, timepart_retro reproduced the race intermittently, with 5 failures across 12 targeted runs.

With the fix:

• timepart_retro: 8/8 passes
• sc_resume_partition: pass
• timepart_events: pass
• sc_timepart: pass
• No cores observed

Diagnostic runs also confirmed that after the fix the old master no longer deletes the sc_list, and the new master consistently finds and resumes it.

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
sc_truncate_multiddl_generated [db unavailable at finish] **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**