One-cluster convergence conjecture for causal attention with a multidimensional dominant eigenspace

About 2 years old · traced to

Let QQ and KK be arbitrary matrices, and let VV be a matrix whose largest eigenvalue λmax⁡>0\lambda_{\max}>0 is real, with eigenspace LL satisfying dim⁡L≥2\mathop{\rm \dim} L\geq2. Assume that Vz∈L⊥Vz\in L^{\perp} and ⟨Vz,z⟩<λmax⁡∣z∣2\langle Vz,z\rangle<\lambda_{\max}|z|^2 for every z∈L⊥z\in L^{\perp}. For an initialization (x1(0),…,xn(0))(x_1(0),\ldots,x_n(0)), let PL⊥\mathbf{P}_{L^{\perp}} denote orthogonal projection onto L⊥L^{\perp}, set y1:=PL⊥(x1(0))y_1:=\mathbf{P}_{L^{\perp}}(x_1(0)), and define ξ:=y1/∣y1∣\xi:=y_1/|y_1|. One-cluster convergence conjecture. For almost every initialization, the causal attention dynamics converge to one cluster, specifically

∀k∈[n],lim⁡t→∞xk(t)=ξ.\forall k\in[n],\qquad \lim_{t\to\infty}x_k(t)=\xi.

This is the proposed description of the positive-dominant-eigenvalue case when the dominant eigenspace has dimension at least two. The paper presents it as a conjectural generalization because critical manifolds make the corresponding stable-manifold analysis technically difficult; no resolution is given.

References

Primary source

Nikita Karagodin, Yury Polyanskiy and Philippe Rigollet, “Clustering in Causal Attention Masking”, arXiv:2411.04990 (2024).

Progress summary

Never refreshed

Nothing recorded yet. Refresh searches the literature and the public web for attempts on this problem, and writes the first summary here.

Solutions 0

No solutions have been posted yet.