4 problems

Stochastic optimization methods such as Adam and Clip-SGD use adaptive stepsizes, and Clip-SGD is known to converge in expectation when the gradient noise has a bounded $\alpha$-th…

Adam's $\ell_\infty$-geometry conjecture. $\mathtt{Adam}$ outperforms $\mathtt{GD}$ due to its utilization of $\ell_\infty$ geometry, under which the loss function could have bette…

When $\theta_t=\theta$ is constant, Adam and RMSProp with $\beta_t=0$ can diverge even in the convex setting. Let $\Gamma_t$ denote the matrix defined in Eq. (Gammat) of the source…

In online convex optimization, data points arrive in streams and the probabilistic model generating them may change over time, so the optimal point $x^{*,t}$ moves during the optim…