1 problem

0 votes0 replies0 views
Conjecture that AMSGrad's maximum operator impedes hyper-gradient flow
The base optimizer AMSGrad uses a maximum operator in its second-moment term, and $\texttt{MADA}\xspace\$ updates optimizer coefficients using hyper-gradients. The hyper-gradient o…
Conjectureopenadaptive-optimizers hypergradients optimizationarXiv posted 2 days ago

Conjecture that AMSGrad's maximum operator impedes hyper-gradient flow