Weak identifiability of MLP parameters under arbitrary activation functions

Let $\sigma$ be an activation function satisfying the mild assumptions in the claim, let $\Theta$ be the parameter space of an MLP, and let $f(\bullet;\theta)$ denote the function represented by a parameter $\theta \in \Theta$. A parameter $\theta' \in \Theta$ is weakly identifiable when it has the weak identifiability property for the MLP parameterization.

Weak identifiability conjecture. With mild assumptions on the activation function $\sigma$, for every parameter $\theta \in \Theta$ of an MLP, there exists a parameter $\theta' \in \Theta$ such that $f(\bullet;\theta)=f(\bullet;\theta')$, and $\theta'$ is weakly identifiable.

For arbitrary activation functions, this is described as a major open problem in the theory of deep learning; the precise mild assumptions and the weak identifiability property are defined elsewhere in the paper.