Cover all ML-DSA rejection-sampling paths in the sigGen CAST

[jakemas]

Context and motivation

FIPS 140-3 IG 10.3.A Note23 recommends the ML-DSA signature generation CAST cover every reachable rejection path in FIPS 204 Algorithm 7: z, r0, the number of 1s in h, and ct0 (ML-DSA-44 only). It points at Table 1 in Section 6.1.2 of draft-celi-acvp-ml-dsa for values.

Our CAST signs a self-chosen message. With counters on the four rejection branches, it only ever takes two:

existing CAST vector   rejections=2  z=1 r0=1 ct0=0 h=0

So ct0 and the OMEGA bound on h are untested, and z/r0 are hit by luck rather than by construction.

Description of changes

Use ML-DSA-44 row 1 of that Table 1 for the keyGen/sigGen CAST vectors. It rejects once at each of the four checks:

new CAST vector        rejections=4  z=1 r0=1 ct0=1 h=1

The row's rnd is all zeros, which the CAST already used, so this is a constant swap: two extra loop iterations, no extra keygen, sign or verify. The keyGen seed changes rather than a second key being derived for signing, because ml_dsa_44_keypair_internal_no_self_test() runs the PCT and a second keygen would roughly triple the CAST.

The draft publishes SHA2-256 of pk || sk and of the signature instead of the values themselves. The expanded constants were generated from the published seed and checked against both digests, which are recorded next to the seed.

Two places keep their own copies of these constants and are updated to match. break-kat.go hardcodes the CAST inputs it corrupts, and without the update break-kat cannot find them and exits 3. fips_callback_test.cc asserts the exact CAST failure message, so it needs the new expected public key and signature; the ML-DSA-sign "Calculated" value also changes, since break-kat zeroes the message and the signing key is now different.

generated-src/crypto_test_data.cc.tar.bz2 is regenerated with util/generate_build_files.py, as the KAT files below are embedded into crypto_test_data.cc.

Finally, add the 27 published vectors (Table 1 and Table 2) to the existing MLDSA_{44,65,87}_ACVP_sigGen.txt files so they run under ACVPSigGen. Those records carry siggen_key_hash and siggen_sig_hash, which ACVPSigGen now checks when present, pinning the ~540KB of expanded hex to NIST's digests.

Testing

All 27 vectors reproduce, including Table 2's exact rejection counts. Worth noting, since we check the ct0 norm before MakeHint rather than after, and the draft warns that such deviations can make these vectors unusable.

ACVPSigGen now runs 22/25/25 records for ML-DSA-44/65/87. Confirmed the added records really execute: corrupting one siggen_sig_hash and rebuilding fails with SHA2-256(sig) mismatch.

• crypto_test: 3038 passed, 2 skipped (environment).
• crypto_test with -DFIPS=1 -DBUILD_SHARED_LIBS=1: 4093 passed, 3 skipped. Exercises the CAST aborting on mismatch, via SelfTests.KAT and FIPSCallback.PowerOnSelfTests.

Static FIPS builds are Linux-only, so tests/ci/run_fips_callback_tests.sh was run on an AL2023 x86_64 host: both build phases, all 39 break-KAT cases and all 5 PWCT cases pass, including ML-DSA-keyGen, ML-DSA-sign, ML-DSA-verify and MLDSA_PWCT. 45 FIPSCallback assertions pass, 0 fail.

Review considerations

No API or ABI change. Touches the FIPS boundary, but only the CAST's constants, not the operations it performs.

Two things left out:

• The ML-DSA CAST covers ML-DSA-44 only, and Note23 also describes coverage for 65 and 87. If one parameter set is considered sufficient because ml_dsa_sign_internal is parameter-set-independent, that is probably worth writing down; otherwise 65/87 need CASTs. The new KATs cover those paths in the test suite, but not as CASTs.
• modulewrapper.cc registers "deterministic": [false]. The draft's rejection-path assurance needs signatureInterface = internal and deterministic = true, so CAVP will not send these cases as registered. processMlDsaSigGen already passes rnd through, but the server may omit rnd for deterministic signing and the wrapper would need to supply zeros.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

[github-actions]

🔒 Security Review — View Report

Please review before merging.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.23%. Comparing base (144a1a3) to head (5ecac50).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3545      +/-   ##
==========================================
+ Coverage   78.22%   78.23%   +0.01%     
==========================================
  Files         700      700              
  Lines      126026   126035       +9     
  Branches    17428    17427       -1     
==========================================
+ Hits        98581    98606      +25     
+ Misses      26573    26559      -14     
+ Partials      872      870       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.