Highway Networks - Deep Neural Network Explained
Highway Networks are a type of network inspired by LSTM that make use of learnable information highway to let inputs flow unimpeded to subsequent layers.
They have a lot of similarities with residual neural networks and offer deep insight into how to make training deeper neural networks possible.
Table of Content
Introduction: 0:00
Degradation Problem: 0:38
Idea Behind Highway Networks: 1:14
Formulas: 2:11
Training & Data: 2:57
Plain VS Highway: 3:34
MNIST Sanity Checks: 4:19
FitNet vs Highway: 4:37
SOTA vs Highway: 5:00
Highway Activation Analysis: 5:26
Highway Ablation Analysis: 7:34
Conclusion: 8:39
Highway Networks Paper: https://arxiv.org/pdf/1505.00387v2
Training Very Deep Network Paper: https://arxiv.org/pdf/1507.06228
For an implementation of Highway Networks do check this repository:
https://github.com/protonx-tf-03-proj...
----
Join the Discord for general discussion: / discord
----
Follow Me Online Here:
Twitter: / yacineaxya
GitHub: https://github.com/yacineMahdid
LinkedIn: / yacinemahdid
___
Have a great week! 👋