When we are dealing with Mean Square Error (MSE) loss function in optimization problems, we often add $L_1$ or $L_2$ penalty terms (or a combination of both) to the MSE loss function while training. When used with MSE loss, these regularisation terms can promote sparsity in the weights or control the magnitude of the weight vector.
Will $L_1$ and $L_2$ regularisation have the same/similar effect when used with other loss functions such as binary cross-entropy loss?
The effects of L1 and L2 regularization on other loss functions, such as binary cross-entropy loss, can be similar in some aspects but may not be exactly the same as when used with mean square error (MSE) loss. Let's briefly recap the characteristics of L1 and L2 regularization: L1 Regularization (Lasso): This term adds the absolute values of the weights to the loss function. It tends to induce sparsity in the weight vectors, meaning that some weights become exactly zero. This can be useful for feature selection. L2 Regularization (Ridge): This term adds the squared values of the weights to the loss function. It tends to penalize large weights and can prevent the weights from becoming too large. When applied to a different loss function, such as binary cross-entropy, the regularization terms still have a similar purpose, but their impact may be expressed differently. Binary cross-entropy is commonly used for binary classification problems. The effect of regularization in this context may include: L1 Regularization: It can still induce sparsity in the weight vectors, promoting some weights to become exactly zero. This can be useful for feature selection even in the context of binary cross-entropy loss. L2 Regularization: It can still control the magnitude of the weight vector, preventing some weights from becoming too large. This can help in preventing overfitting by discouraging the model from relying too heavily on a small subset of features.
