Regularization in Machine and Deep Learning (Ridge, Lasso, and Layer weight regularizers)
This video explains the basic idea behind regularization in machine learning and deep learning. The idea is to constrain complex machine learning models to perform better on test/unseen data sets, i.e., improve generalization. In other words, the goal of regularization is to avoid overfitting. To this end, this video discusses how to modify cost or loss functions for both classification and regression problems for the purposes of regularization. The regularization parameter is a constant in the “penalty” term added to the cost function, which should be fine tuned to find better tradeoffs between bias and variance. This video discusses both Ridge regression (sklearn.linear_model.Ridge) and LASSO in scikit-learn (sklearn.linear_model.Lasso). This tutorial also discusses the concept of regularization for support vector machines: the Regularization parameter (often termed as C parameter in Python's scikit-learn library) tells the SVM optimization how much to avoid misclassifying training data points. Finally, we discuss regularization for neural networks or deep learning, which is known as layer weight regularizers in Keras. Particularly, we use kernel_regularizer to apply a penalty on the layer's kernel. Weight decay works by adding a penalty term to the cost function of a neural network which has the effect of shrinking the weights.
#Regularization #Overfitting #Penalty