Why limiting weights help against overfitting in neural networks? I have read some stuff about regularization but cannot understand it yet. It is said that smaller weights favor a prior information of weights being distributed around zero. But why it should be true? This prior by itself does not have anything to do with overfitting.
It is also said that regularization reduces (and controls) the network capacity and thus reduces the chance of overfitting. Yes by regularization we limit the network and it can not be matched exactly to the training signal. But then how it comes that such a limitation could improve generalization? I do not find a logical relation between these two points. A network that is weak in matching the training data is also probable (maybe to a more degree compared to a higher capacity network) to fail on non-seen data.
And when we limit the weights we work around zero, exactly in the linear region of sigmoid function. All of the claims about power of neural network comes from the activation function being nonlinear. If we are working on the linear region, where is that benefit?
 A: I see two points of confusion. First, to predict that poor training recovery means poor prediction assumes that the training set is a representative sample of the entire data space. In high dimensional data, this is often not the case. So relaxing fit to the training data tends improve fit to examples not included in the training. I meant representative of data space, not of the samples. Consider the case where you training set consists primarily of common cases, a few uncommon ones, and several missing cases. Then your model will only perform well on that small portion of the data space; it is over-fit. This is a result of the optimization process, which tries to match the exact data as closely as possible. Between weight sets that give similar predictions for the data, optimization will always favor weights that match the data best, even while others may give better prediction on new data. If your data set is very large and/or highly redundant, then regularization may not improve model accuracy and/or may not shrink the weights very much. 
Second, while regularization does move all weights towards zero, it does so in competition with model accuracy. Therefore, at any point in the network, important features will not have their weights shrunk as much as unimportant ones. As long as some weights are still bounded away from zero, then the total activation will still be highly non-linear. 
