Overparameterization Improves Optimization for Neural Operators

Let a neural operator have adjustable capacity, for example through its number of layers or neurons, and consider the non-convex optimization landscape of its training loss. Overparameterization improves optimization. Increasing the capacity of neural operators can improve the optimization landscape by reducing the prevalence of poor local minima and saddle points, thereby enhancing the convergence of gradient-based methods. This claim concerns optimization rather than approximation error: it proposes that larger neural operators may be easier to train despite the non-convexity of their loss landscapes. The source provides no theorem or evidence establishing the claim in the stated generality.