Post-LayerNorm smuggled-bias degeneracy conjecture

Because the outputs are constrained by $\mathbf{1}^T x=1$, let $x'=x-\frac{1}{d}\mathbf{1}$, so that $\mathbf{1}^T x'=0$, and let $W\in\mathbb{R}^{m\times d}$ act on the input. Define the smuggled bias by

$$b_{\mathrm{smuggled}}=\frac{1}{d}W\mathbf{1}.$$

Then $Wx=Wx'+b_{\mathrm{smuggled}}$. Post-LayerNorm smuggled-bias degeneracy conjecture. For $y=\mathrm{LN}(Wx)$ with mean-squared-error loss on the LayerNorm output, the LLC drop relative to a full-rank Gaussian baseline is

$$\Delta\lambda=\frac{m}{2}.$$

LayerNorm's translation invariance should make $b_{\mathrm{smuggled}}$ an effective blind spot, although the precise symmetry group of the Post-LayerNorm loss landscape has not been characterized. Current SGLD-based LLC estimates are unreliable in this setting, with the reported estimate $\Delta\lambda=-0.92\pm0.86$ over five seeds; a direct proof or a better-calibrated estimator is still needed.