Jichu Li* - October 19, 2026
<aside> đź”–
TL;DR A spectral advantage survives after we remove the Transformer and train just one matrix on real next-token statistics. Two ingredients matter together: unequal context frequencies and structured context–target pairings. Supplying the unigram background strengthens the separation; correcting input-side geometry can make ordinary gradient descent faster still.
</aside>
Language modeling starts with a simple task: predict what comes next. Modern pretraining wraps it in attention, feed-forward layers, residual connections, normalization, and learned representations. The Transformer makes these components work together remarkably well—but makes it harder to isolate why one optimizer works better than another. Vaswani et al., 2017
Muon makes that question especially interesting. Its matrix updates reshape the singular values of a momentum matrix, with empirical successes extending to large language-model training. Without momentum, its idealized update uses the gradient's polar factor. This is a geometric choice: “steepest descent” depends on how we measure a step, and coordinatewise and spectral geometries need not favor the same directions. Spectral normalization therefore asks a different question from simply taking the sign of each matrix entry. Jordan, 2024; Liu et al., 2025; Bernstein and Newhouse, 2024
Several clues point toward the data. Heavy-tailed class imbalance separates the learning speeds of frequent and rare classes; minimal models yield provable benefits for sign descent. Bigram and associative-memory studies connect optimizer behavior to frequency, representation capacity, and learning in the tail. These clues motivate a closer look at the statistics behind the architecture. Kunstner et al., 2024; Yadav et al., 2026; Kunstner and Bach, 2025; Wang et al., 2026; Li et al., 2026; Kim et al., 2026
Real text brings two ingredients together: some contexts occur much more often, and different contexts predict different next tokens. A frequency distribution describes the first ingredient, but misses the second. A common context and a rare one need not pose the same prediction problem. Could their interaction already explain a preference for spectral updates?
<aside> 🔍
Our question: Strip away the architectural complexity of LLM pretraining and keep only next-token data statistics. Can real text still favor spectral geometry?
</aside>
We will shrink the model to one trainable matrix, give every direction exact line search, and change one statistical ingredient at a time. The aim is an empirical dissection: what sustains the separation, what weakens it, and where does the extra progress go?
How much of pretraining can we remove while keeping its prediction task? Start with the full prefix, shorten it to one token, freeze the representations, and leave just one matrix to learn:
$$ \begin{aligned} \text{prefix + Transformer}&\longrightarrow\text{one-token context}\\ &\longrightarrow\text{frozen representations}\\ &\longrightarrow\text{one trainable matrix }\mathbf W. \end{aligned} $$
From the training text, collect adjacent pairs $(x_t,x_{t+1})$. If $n_{xy}$ counts how often token $y$ follows $x$, $n_x=\sum_y n_{xy}$, and $T=\sum_x n_x$, then the two statistics we retain are
$$ \underbrace{\pi_x=\frac{n_x}{T}}{\text{how often the context occurs}}, \qquad \underbrace{p(y\mid x)=\frac{n{xy}}{n_x}}_{\text{what tends to follow it}}. $$
Frozen normalized random vectors $\mathbf e_x,\mathbf u_y\in\mathbb R^d$ turn these empirical bigrams into a small matrix optimization problem:
$$ \begin{aligned} q_{\mathbf W}(y\mid x)&=\operatorname{softmax}y(\mathbf u_y^\top\mathbf W\mathbf e_x+b_y),\\ \mathcal L(\mathbf W)&=-\sum_x\pi_x\sum_y p(y\mid x)\log q{\mathbf W}(y\mid x). \end{aligned} $$
Write $\mathbf G_t=\nabla\mathcal L(\mathbf W_t)$. Our three directions differ only in how they transform this gradient:
$$ \begin{aligned} \mathbf D_t^{\mathrm{GD}}&=-\mathbf G_t,\\ \mathbf D_t^{\mathrm{SignGD}}&=-\operatorname{sign}(\mathbf G_t),\\ \mathbf D_t^{\mathrm{Spectral}}&=-\operatorname{polar}(\mathbf G_t). \end{aligned} $$
Here $\operatorname{polar}(\mathbf G)=\mathbf P\mathbf Q^\top$ for $\mathbf G=\mathbf P\boldsymbol\Sigma\mathbf Q^\top$. Spectral uses the exact SVD without momentum. Every direction gets its own numerically implemented exact line search:
$$ \alpha_t\in\arg\min_{\alpha\ge0}\mathcal L(\mathbf W_t+\alpha\mathbf D_t), \qquad \mathbf W_{t+1}=\mathbf W_t+\alpha_t\mathbf D_t. $$