1 problem

0 votes0 replies2 views
Conjecture on the cause of frequent clipping in distributed gradient clipping
The algorithm operates on local gradients computed on each GPU, without averaging those gradients across all machines before clipping. Conjecture. The more frequent clipping observ…
Conjectureopendeep-learning distributed-optimization gradient-clippingarXiv posted 2 days ago

Conjecture on the cause of frequent clipping in distributed gradient clipping