Conjecture that nonlinear neural-network loss functions are not rotationally invariant

Let $csigma$ be a smooth activation function that is not linear, let the data set contain at least two data points, and let $L$ be the associated loss function. Let ${\operatorname{Rot}}$ denote the class of rotationally invariant functions.

Rotational-invariance conjecture. If the activation function $csigma$ is a smooth function that is not linear and the data set contains $cgeq 2$ data points, then

$$L \notin {\operatorname{Rot}}.$$

This conjecture concerns the geometric symmetries of loss functions for nonlinear networks. It appears in the paper's discussion of general expectations, and the supplied text provides no resolution beyond posing the claim.