Adam Optimization from Scratch in Python
Adam is yet another stochastic gradient descent technique, building on Adadelta and RMSProp it fixes the shortcoming of Adagrad by using two running average in its calculation.
You can find the code for this video over here: https://github.com/yacineMahdid/artif...
Credit
Check out this blogpost for more gradient descent explanation: https://ruder.io/optimizing-gradient-...
The music is taken from Youtube music!
Table of Content
Introduction: 0:00
Theory: 0:21
Python Implementation: 3:49
Conclusion: 12:04
Here is an explanation of Adam from the blog post mentioned above which I find very intuitive:
"Adaptive Moment Estimation (Adam) is another method that computes adaptive learning rates for each parameter. In addition to storing an exponentially decaying average of past squared gradients vt like Adadelta and RMSprop, Adam also keeps an exponentially decaying average of past gradients mt, similar to momentum. Whereas momentum can be seen as a ball running down a slope, Adam behaves like a heavy ball with friction, which thus prefers flat minima in the error surface. "
Reference
Kingma, D. P., & Ba, J. L. (2015). Adam: a Method for Stochastic Optimization. International Conference on Learning Representations, 1–13
----
Join the Discord for general discussion: / discord
----
Follow Me Online Here:
Twitter: / codethiscodeth1
GitHub: https://github.com/yacineMahdid
LinkedIn: / yacine-mahdid-809425163
Instagram: / yacine_mahdid
___
Have a great week! 👋