Jupyter Notebook

Adaptive Gradient Descent (AdaGrad)

Equation:

$\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\frac{\eta}{\sqrt{\sum\limits_{i=1}^t\mathbf{g}^{(t)T}\mathbf{g}^{(t)}+\varepsilon}}\mathbf{g}^{(t)}$

$\mathbf{g}^{(t)}=∇_\mathbf{w}L(\mathbf{w}^{(t)})\\$

Parameters:

• $\varepsilon$ is a constant that make sure the denominator is not zero (default in Tensorflow/Keras: $1e-7$).
• $\eta$ is the learning rate (default in Tensorflow/Keras: $0.001$)

Properties:

• The magnitude of the update will always be smaller in later iterations.
• The update become very inefficient in later iterations.
• Vulnerable to local minimum.
• Converge even with large learning rate.

API:

opt = tf.keras.optimizers.Adagrad(learning_rate)
opt.minimize(loss, var_list=[w])

RMSprop

Equation:

$\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\frac{\eta}{\sqrt{G^{(t)}} + \varepsilon}\mathbf{g}^{(t)}$

$\mathbf{g}^{(t)}=\nabla_\mathbf{w}L(\mathbf{w}^{(t)})$