Hertrich–Lindner–Raginsky–Bach depth-separation conjecture for ReLU networks

Let $\operatorname{ReLU}_n(k)$ denote the set of continuous piecewise linear functions defined on $\mathbb{R}^n$ and computable with $k$ hidden layers. Hertrich–Lindner–Raginsky–Bach's depth-separation conjecture.

$$\operatorname{ReLU}_n(k-1)\subsetneq\operatorname{ReLU}_n(k)\quad\text{for all }k\leq \lceil\log_2(n+1)\rceil.$$

The conjecture asserts that the logarithmic upper bound on the number of hidden layers needed to represent every continuous piecewise linear function is necessary. It remains open for every input dimension $n\geq 4$; in particular, no continuous piecewise linear function is known to require more than two hidden layers.