Stochastic Attention via Langevin Dynamics on the Modern Hopfield Energy
创建于 更新于
摘要
This paper shows that a standard attention head implements gradient descent on the modern Hopfield log-sum-exp energy and that applying Langevin dynamics to that energy yields a training-free stochastic attention sampler controlled by a single inverse-temperature parameter β; the sampler interpolates between exact retrieval and open-ended generation, admits convergence guarantees due to the Hopfield energy’s analytic structure, and—via an SNR-based temperature rule—yields substantially higher novelty and diversity than learned baselines on image and financial pattern tasks [page::0][page::8].
