ReLU network approximation error conjecture

Let $4\Theta4$ be the input domain and let $f \in L^2(\Theta)$ satisfy $|| f ||_{\infty} < \infty$. Let $f_{\Phi}$ be the output of a ReLU network with trainable parameters $\Phi$, consisting of $L$ layers and $K$ neurons in each layer. Then there is a constant $C>0$ such that

$$\min_{\Phi} || f - f_{\Phi} ||_{L^2(\Theta)}^2 \le C \, \frac{|| f ||_{\infty}^2}{K L} + {\mathcal O}(K^{-2} + L^{-4}).$$

ReLU approximation error conjecture. Under these assumptions, the displayed bound holds.

This conjecture is presented as an analogue for ReLU networks of a previously established result for Fourier-feature residual networks. It is intended to support complexity comparisons for residual multi-fidelity neural-network surrogates, but the source gives no proof or resolution.