I frequently hear some very conflicting claims regarding deep learning algorithms. Currently, I am a bit confused on the role of regularization. I have listed my queries below regarding regularization and all queries are inter-related to each other:
We use regularization in deep neural nets to make generalization error small (since overfitting leads to an increase in generalization error). So, basically regularization is used to control overfitting. Does overfitting means the optimization/training algorithm gets stuck in local minima?
When we use weight decay, it penalizes the large weights and direct the weights towards smaller values. When the values of weights become very small, doesn't it hinder in learning process or makes the optimization algorithm to stuck in local minima? If the algorithm is not learning because of smaller weights, can we say its stuck in local minima?
Do the regularization strategies make the convergence of optimization algorithm faster? If yes, how do they?