Adam: A Method for Stochastic Optimization

Adam combines adaptive learning rate methods with momentum-based optimization. It maintains exponential moving averages of both gradients and squared gradients, with bias correction for stability. Computationally efficient and invariant to diagonal rescaling, Adam became the default optimizer in modern deep learning.