Effective Lipschitz concentration for transformer language models

Let $F_W$ be a transformer trained to near-zero cross-entropy loss on a corpus with entropy $H$, and let $P_{\mathcal{M}}$ be the input distribution. Define

$$\Lambda_{\mathrm{eff}}=\mathbb{E}_{x\sim P_{\mathcal{M}}}\left[\sup_{x':\|E(x')-E(x)\|\leq\varepsilon}\frac{\|F_W(x')-F_W(x)\|}{\|E(x')-E(x)\|}\right].$$

Effective Lipschitz concentration. The effective Lipschitz constant satisfies $\Lambda_{\mathrm{eff}}\ll\Lambda(W)$, specifically $\Lambda_{\mathrm{eff}}=O(\mathrm{poly}(H))$, while $\Lambda(W)$ can grow exponentially in $L$.

The claim is supported by empirical generalization to semantically similar inputs and by the approximation of the conditional distribution $P_{\mathcal{M}}(\cdot\mid x)$ at near-zero cross-entropy loss. A rigorous proof would require bounds on the variation of this conditional distribution under embedding perturbations, depending on the model's local spectral properties; this remains open.