I have read some stuff about regularization but cannot understand it yet. It is said that smaller weights favor a prior information of weights being distributed around zero. But why it should be true? This prior by itself does not have anything to do with overfitting.
It is also said that regularization reduces (and controls) the network capacity and thus reduces the chance of overfitting. Yes by regularization we limit the network and it can not be matched exactly to the training signal. But then how it comes that such a limitation could improve generalization? I do not find a logical relation between these two points. A network that is weak in matching the training data is also probable (maybe to a more degree compared to a higher capacity network) to fail on non-seen data.
And when we limit the weights we work around zero, exactly in the linear region of sigmoid function. All of the claims about power of neural network comes from the activation function being nonlinear. If we are working on the linear region, where is that benefit?