Existence of double scaling limit for random neural networks

Consider a random depth $L$ neural network with input dimension $n_0$, hidden layer widths

$$n_1,\ldots,n_L=n\gg 1,$$

output dimension $n_{L+1}$ and non-linearity $\sigma$. Suppose that the network is tuned to criticality, meaning that the criticality condition is satisfied. Fix a non-zero network input $x_\alpha\in\mathbb R^{n_0}$ and write $\xi=L/n$. For each $k\geq 1$, let $\kappa_{2k;\alpha}^{(L)}$ denote the corresponding even cumulant and let $K_{2;\alpha}^{(L)}$ denote the corresponding second moment. Existence of the double scaling limit. For each $k\geq 1$ there exists $C_{2k}>0$, depending on the universality class of $\sigma$, such that

$$\frac{\kappa_{2k;\alpha}^{(L)}}{\left(K_{2;\alpha}^{(L)}\right)^k}=C_{2k}\xi^{k-1}+O\left(\xi^k\right).$$

Moreover, for each $\xi\in[0,\infty)$ there exists a probability distribution $\mathbb P_{\xi,\sigma}$ on $\mathbb R$, depending only on $\xi$ and $\sigma$, such that in the double scaling limit

$$n,L\longrightarrow\infty,\qquad \frac{L}{n}\longrightarrow\xi,$$

the random variable $z_{i;\alpha}^{(\ell)}$ converges in distribution to a random variable with law $\mathbb P_{\xi,\sigma}$. This conjecture proposes a universal limiting description of critically tuned random neural networks when width and depth grow proportionally; the existence of the limiting distribution and the stated cumulant asymptotics are not established in the supplied text.