Ali Ghodsi, Deep Learning, Dropout, Batch Normalization, Fall 2023, Lecture 5
Dropout, Batch normalization
Batch normalization was initially inspired by the notion of internal covariate shift (ICS). However, it's now understood that Batch normalization doesn't address ICS, and we also recognize that ICS doesn't have detrimental effects. Instead, Batch normalization reduces Lipschitzness and enhances the smoothness of the loss function. This video succinctly covers dropout and Batch normalization and briefly elucidates why they are effective.