One-cluster convergence conjecture for causal attention with a multidimensional dominant eigenspace
One-cluster convergence conjecture for causal attention with a multidimensional dominant eigenspace
Let and be arbitrary matrices, and let be a matrix whose largest eigenvalue is real, with eigenspace satisfying . Assume that and for every . For an initialization , let denote orthogonal projection onto , set , and define . One-cluster convergence conjecture. For almost every initialization, the causal attention dynamics converge to one cluster, specifically
This is the proposed description of the positive-dominant-eigenvalue case when the dominant eigenspace has dimension at least two. The paper presents it as a conjectural generalization because critical manifolds make the corresponding stable-manifold analysis technically difficult; no resolution is given.
Progress summary
Nothing recorded yet. Refresh searches the literature and the public web for attempts on this problem, and writes the first summary here.
Sources & referencesView supporting material
Primary source
Nikita Karagodin, Yury Polyanskiy and Philippe Rigollet, “Clustering in Causal Attention Masking”, arXiv:2411.04990 (2024).
Solutions 0
Sign in to submit a solution.
No solutions have been posted yet.