Convergence conjectures for deep linear neural network gradient flows

Assume that $X X^T$ has full rank. Let $N$ be the depth, let $W(t)=W_N(t)\cdots W_1(t)$ denote the product along the gradient flow, and let $\mathcal{M}_k$ denote the rank-constrained manifold used in Theorem 1.1. In the autoencoder case, assume $X=Y$, $d=d_0=d_N$, let $r=\min_{i=1,\ldots,N}d_i$, and let $\lambda_1\geq\cdots\geq\lambda_d$ be the eigenvalues of $X X^T$ with corresponding orthonormal eigenvectors $u_1,\ldots,u_d$. Write $U_r$ for the matrix whose columns are $u_1,\ldots,u_r$. Convergence conjecture. (a) The statement in Theorem 1.2(b) also holds for $N>2$. (b) If $\lambda_r>\lambda_{r+1}$, $W(0)U_r$ has rank $r$, and $u_i^T W(0)u_i>0$ for every $i\in\{1,\ldots,r\}$, then $W(t)$ converges to $\sum_{i=1}^r u_i u_i^T$. (c) In Theorem 1.3, if $k\leq\operatorname{rank}(Q)$, convergence to a critical point on some $\mathcal{M}_{\ell}$ with $\ell<k$ occurs only for a measure-zero set of initial values. These claims would sharpen the generic convergence theory for deep linear networks, extending the established depth-two result and describing the limiting autoencoder and lower-rank behavior. Their resolution is not supplied in the source.