What Really Matters Inside Muon

What is Muon?

Optimization in deep learning has evolved in stages. SGD updates parameters using the raw gradient:

$$W_{t+1} = W_t - \eta \nabla W_t$$

where we use $\nabla W_t$ as shorthand for $\nabla_{W_t} \mathcal{L}(W_t)$, the gradient of the loss with respect to the weight matrix.

SGD treats every coordinate equally. While it is simple and stable, it is sensitive to conditioning — if some parameters have much larger gradients than others, a single learning rate either undershoots the small gradients or overshoots the large ones. To solve this, Adam rescales each coordinate adaptively:

$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$ $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$ $$W_{t+1} = W_t - \eta \frac{m_t}{\sqrt{v_t} + \epsilon}$$

The key idea is that dividing by $\sqrt{v_t}$ normalizes each coordinate by its own scale. Parameters with consistently large gradients get divided by a large number, and parameters with small gradients get a relative boost. This makes the effective step size roughly uniform across coordinates, regardless of how different the raw gradient magnitudes are.

Most models use some version of Adam for pretraining. But as models get larger, we need a higher-efficiency optimizer. The core insight behind Muon is to extend this same normalizing principle from individual coordinates to the singular values of the gradient matrix. Instead of treating the gradient as a flat vector of independent coordinates, Muon treats it as a matrix and asks: what if we normalize across the spectral structure of the gradient, not just its entries?

How Does Muon Work?

Let $G_t = \nabla W_t \in \mathbb{R}^{m \times n}$ denote the gradient of a weight matrix at step $t$. Muon performs a spectral transformation of the update matrix rather than applying element-wise scaling.

Compute the singular value decomposition: $G_t = U \Sigma V^\top$, where

$$\Sigma = \operatorname{diag}(\sigma_1, \dots, \sigma_r)$$

A mapping function $y : \mathbb{R}_{\ge 0} \to \mathbb{R}_{\ge 0}$ is applied to the singular values:

$$\Sigma' = \operatorname{diag}(y(\sigma_1), \dots, y(\sigma_r))$$

The transformed update becomes $\Delta W_t = U \Sigma' V^\top$ and the parameter update is $W_{t+1} = W_t - \eta \Delta W_t$.

The behavior of Muon is entirely determined by the choice of $y(\sigma)$. In the standard Muon implementation, $y(\sigma) = 1$ for all $\sigma$, meaning every singular value is mapped to the same constant. This corresponds to replacing the gradient with its orthogonal polar factor $P = U V^\top$.

Newton–Schulz Iteration

Computing full SVDs at each step is expensive. In practice, Muon uses a Newton–Schulz iteration to approximate the polar factor $U V^\top$ directly, without ever computing individual singular values.

Given a normalized matrix $X_0$, iterate:

$$X_{k+1} = \frac{1}{2} X_k (3I - X_k^\top X_k)$$

Under appropriate spectral radius conditions, this converges quadratically to the orthogonal polar factor $X_\infty = U V^\top$, making spectral normalization computationally viable at scale.

Affine Sweep Experiments

Evaluating an affine mapping $y(\sigma) = (1-c)\sigma + c$ reveals that once $c$ exceeds a small threshold (approximately $0.1$), performance stabilizes. A pure clamping function $y(\sigma) = \max(0.1, \sigma)$ performs virtually identically to full polar normalization, showing that avoiding spectral collapse in the weak gradient directions is what truly drives stability.