Steady-state convergence and moment conjecture for general-objective SGD

Let $f:cmathbb{R}ctocmathbb{R}$ be convex and of class $C^h(cmathbb{R})$ for an even integer $h>0$, with unique minimizer $x^*$, satisfying $\lim_{x\to x^*}f^{(k)}(x)=0$ for $1\leq k\leq h-1$, $f^{(h)}(x^*)>0$, and uniformly bounded $f^{(h+1)}$. Let the noise sequence $\{\xi_k\}_{k\geq0}$ be i.i.d. with $\mathbb{E}[\xi_k^3]<\infty$, and let $\{X_k^{(\alpha)}\}_{k\geq0}$ denote the SGD iterates with stepsize $\alpha$. Steady-state convergence and moment conjecture. Under these assumptions, there exists $a>0$ such that, for every $\alpha\in(0,a)$, $\{X_k^{(\alpha)}\}_{k\geq0}$ converges in distribution as $k\to\infty$ to a random variable $X^{(\alpha)}$. Moreover, there exists $C_h>0$ such that

$$\sup_{\alpha\in(0,a)}\mathbb{E}\lvert X^{(\alpha)}-x^*\rvert^{3h}\leq C_h.$$

This conjecture extends steady-state stability beyond strongly convex and smooth objectives to objectives with higher-order local growth near their minimizer; its status is not established in the supplied text.