Post-LayerNorm smuggled-bias degeneracy conjecture
Because the outputs are constrained by 1Tx=1\mathbf{1}^T x=11Tx=1, let x′=x−1d1x'=x-\frac{1}{d}\mathbf{1}x′=x−d11, so that 1Tx′=0\mathbf{1}^T x'=01Tx′=0, and let W∈Rm×dW\in\mathbb{R}^{m\times d}W∈Rm×d act on the input. Def…