Weak identifiability of multi-head attention parameters

Let $\Theta$ be the parameter space of a multi-head attention (MHA) model with value dimension $d_v$ and $H$ heads. Let $S_H$ be the permutation group on the heads, and let

$$K_i=(\operatorname{GL}(d_v)^H)\rtimes S_H.$$

A parameter is weakly identifiable with respect to $K_i$ when it is weakly identifiable after accounting for this symmetry group.

MHA weak identifiability conjecture. For every parameter $\theta\in\Theta$ of an MHA model, the parameter $\theta$ is weakly identifiable with respect to the symmetry group

$$K_i=(\operatorname{GL}(d_v)^H)\rtimes S_H.$$

The conjecture asserts that the parameterization map is well behaved after quotienting by the value-output general linear transformations and head permutations. The source presents identifiability for attention mechanisms as a nascent field and motivates omitting query and key symmetries because equivariance is typically mediated through the value-output path or positional encoding.