One-cluster convergence conjecture for causal attention with a multidimensional dominant eigenspace

From papers

Let QQ and KK be arbitrary matrices, and let VV be a matrix whose largest eigenvalue λmax>0\lambda_{\max}>0 is real, with eigenspace LL satisfying dimL2\mathop{\rm \dim} L\geq2. Assume that VzLVz\in L^{\perp} and Vz,z<λmaxz2\langle Vz,z\rangle<\lambda_{\max}|z|^2 for every zLz\in L^{\perp}. For an initialization (x1(0),,xn(0))(x_1(0),\ldots,x_n(0)), let PL\mathbf{P}_{L^{\perp}} denote orthogonal projection onto LL^{\perp}, set y1:=PL(x1(0))y_1:=\mathbf{P}_{L^{\perp}}(x_1(0)), and define ξ:=y1/y1\xi:=y_1/|y_1|. One-cluster convergence conjecture. For almost every initialization, the causal attention dynamics converge to one cluster, specifically

k[n],limtxk(t)=ξ.\forall k\in[n],\qquad \lim_{t\to\infty}x_k(t)=\xi.

This is the proposed description of the positive-dominant-eigenvalue case when the dominant eigenspace has dimension at least two. The paper presents it as a conjectural generalization because critical manifolds make the corresponding stable-manifold analysis technically difficult; no resolution is given.

Progress summary

Nothing recorded yet. Refresh searches the literature and the public web for attempts on this problem, and writes the first summary here.

Sources & referencesView supporting material

Primary source

Nikita Karagodin, Yury Polyanskiy and Philippe Rigollet, “Clustering in Causal Attention Masking”, arXiv:2411.04990 (2024).

Solutions 0

No solutions have been posted yet.