Jichu Li - October 13, 2026
<aside>
TL;DR
Building on Kim et al. (2026), we further explore how geometry and learning rates shape Gaussian associative-memory learning. We establish one-step optimality of exact polar descent among directions constructed from the initial gradient and output embeddings. We then show why effective late convergence requires an appropriate learning-rate policy, and ask how far scalar tuning can take both GD and spectral descent.
</aside>
Contents
Why study associative memory?
Imagine training a model to answer queries such as a country's capital or a person's attribute. Associative memory isolates the problem of storing and retrieving these associations. The idea runs from Hopfield's (1982) content-addressable networks through dense memories to modern Hopfield models connected to attention. It also helps explain language models: Geva et al. (2021) interpret feed-forward layers as key-value memories, Meng et al. (2022) edit factual associations in weights, and Bietti et al. (2023) use a memory viewpoint to study how Transformer mechanisms emerge during training.
The optimization challenge is that some associations appear constantly while others are rare, and their representations overlap. Studies of long-tail knowledge and knowledge capacity connect factual learning to training exposure. Random-embedding models make frequency imbalance and interference tractable (Cabannes et al., 2024); suitable $d\times d$ memories can store nearly a parameter-count number of associations, up to logarithmic factors (Nichani et al., 2024). A good matrix may therefore exist even when rare associations remain difficult to learn. This leads to our question: with representations fixed, how does an optimizer write associations into its parameters and progress from common ones to rare ones?
What earlier optimizer comparisons tell us
Capacity asks whether a good matrix exists. Optimization asks whether a trajectory finds one quickly. Cabannes, Şimşek, and Bietti (2024) study GD with fixed embeddings and cross-entropy, showing how unequal frequencies, correlated representations, and learning rates affect its dynamics. This motivates studying capacity, update geometry, and learning-rate policy together.
Recent work makes the optimizer comparison explicit. Wang et al. (2025) analyze GD, SignGD, and Muon with orthogonal embeddings and two frequency groups, showing how spectral updates can balance learning across groups. Li et al. (2026) study GD and Muon dynamics with orthogonal embeddings and hierarchical frequencies; their noiseless polynomial-versus-exponential comparison uses a fixed learning rate. In an adjacent model, Kunstner and Bach (2025) derive GD and sign-descent scaling laws for a one-hot bigram problem with square loss.
These papers explain why frequency imbalance and update geometry matter. The Gaussian setting studied here adds many overlapping associations in a fixed-dimensional representation, and it lets us ask which differences survive when scalar learning rates are tuned.
A direct comparison with Kim et al.
Kim et al. (2026) are our closest starting point. Their one-step result already identifies the strong spectral exponent in Gaussian memory. We examine exact-polar directions, rigorous first-gradient information limits, and actual-gradient behavior under explicitly distinguished learning-rate policies.
| Question | Kim et al. | This post's research results |
|---|---|---|
| One-step accuracy | Theorem 4.1 and Corollary 4.2 analyze a stabilized spectral map. Their theorem assumes $N\gtrsim d^{2\alpha+2}$. | The same polynomial loss exponent for exact polar under $d^2\le N\le d^\beta$, plus a matching lower bound for prescribed first-gradient direction rules. |
| Information optimality | Section 5.1 gives a heuristic optimality argument; its equivariance reduction is formal. | The lower bound specifies what the direction may observe and permits arbitrary scalar tuning. |
| Multiple steps | Theorems 5.4–5.5 use thresholded gradients; the actual-gradient Muon extension is conjectured. | Actual-gradient theorems: two-step GD obstruction, capped-GD lower bound, adaptive-GD upper bound, and inverse-time exact-polar lower bound. |
| Fast late convergence | The surrogate recursion motivates rapid progress. | The optimal $\Theta(\log\log d)$ rate remains a conjecture, including its lower bound. |
How to read the rest of the post
We first establish the loss floor and the norm radius needed to approach it. These facts explain why spectral geometry has an advantage in a standard Frank–Wolfe certificate. We then examine actual updates at two accuracies: an early target that one polar step can cross, and a late target near the best possible loss.
We model this competition between useful signal, interference, and frequency using Gaussian query and output embeddings.
Fixed embeddings, a trainable matrix