Fix all-NaN output for complex inputs with 'schatten' preconditioning in Muon - #1761
Open
shoemoney wants to merge 1 commit into
Open
Fix all-NaN output for complex inputs with 'schatten' preconditioning in Muon#1761shoemoney wants to merge 1 commit into
shoemoney wants to merge 1 commit into
Conversation
Author
|
CI note: Build and check types with pyrefly failure is pre-existing on main and unrelated to this PR's changed files. The 10 pyrefly errors are in linear_algebra.py/linesearch.py/_make_pert.py/_projections.py with no overlap with _muon.py/_muon_test.py and reproduce identically on base commit 0391582 (main HEAD). No code fix required from this PR; rebase or label will clear it. |
…ex inputs The modified first Newton-Schulz step for 'schatten' preconditioning computes the Gram matrix as a = x @ x.T, which for complex x is not the Hermitian Gram matrix, so the iteration diverges and orthogonalize_via_newton_schulz returns an all-NaN array for complex parameters. Any muon(preconditioning='schatten') step on a complex model NaN-poisons the parameters on the first update. Same file, same function family: _aol_first_newton_schulz_iteration (line 217, a = x @ x.T.conj()) and _base_newton_schulz_iteration (line 247, a = x @ x.T.conj()) both take the conjugate; schatten, added alongside aol in PR google-deepmind#1602, is the only one that omitted it. .conj() is a no-op for real dtypes, so real-input behavior is unchanged. test_newton_schulz is now parameterized over frobenius, aol, and schatten so the existing real-orthogonality and complex-unitarity assertions cover all three paths; the schatten complex case fails with all-NaN output before this fix and passes after.
shoemoney
force-pushed
the
fix-muon-schatten-complex-conj
branch
from
September 1, 2026 21:15
7de8c7e to
2bc18c1
Compare
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
This looks right to me. The Schatten first step now builds the Hermitian Gram matrix like the AOL/base paths, while .conj() is a no-op for real inputs. Parameterizing the existing orthogonality/unitarity test also covers the missing complex Schatten case. I don't see a blocker here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_schatten_first_newton_schulz_iterationcomputes the Gram matrix asa = x @ x.T(optax/contrib/_muon.py:233on main). For complexxthis is not the Hermitian Gram matrix, so the modified first Newton-Schulz step diverges andorthogonalize_via_newton_schulz(..., preconditioning='schatten')returns an all-NaN array. Amuon(preconditioning='schatten')step on a complex-valued model therefore NaN-poisons the parameters on the first update. Complex support is a tested contract of this code path:test_newton_schulzasserts complex Newton-Schulz produces a unitary matrix, but only exercised the default preconditioning.The siblings in the same function family already take the conjugate:
_aol_first_newton_schulz_iteration(optax/contrib/_muon.py:217) and_base_newton_schulz_iteration(optax/contrib/_muon.py:247) both usea = x @ x.T.conj(). Schatten, added alongside aol in #1602, is the only one that omitted it..conj()is a no-op for real dtypes, so real-input behavior is unchanged.Fix: use
x @ x.T.conj()in the schatten first iteration. Test:test_newton_schulzis now parameterized over'frobenius','aol', and'schatten', so its existing real-orthogonality and complex-unitarity assertions cover all three modified first steps.Verification (CPU, jax 0.11.1): before the fix, the schatten complex case fails with an all-NaN Gram matrix; after,
optax/contrib/_muon_test.pypasses (34 tests, 18 subtests), and the schatten complex output is unitary to ~1e-7 while the real schatten path is numerically unchanged (max deviation from identity ~1e-8).'spectral'is deliberately not added to the parameterization: it shares_base_newton_schulz_iteration, which already conjugates, so it has no instance of this bug.