2
$\begingroup$

I have read some stuff about regularization but cannot understand it yet. It is said that smaller weights favor a prior information of weights being distributed around zero. But why it should be true? This prior by itself does not have anything to do with overfitting.

It is also said that regularization reduces (and controls) the network capacity and thus reduces the chance of overfitting. Yes by regularization we limit the network and it can not be matched exactly to the training signal. But then how it comes that such a limitation could improve generalization? I do not find a logical relation between these two points. A network that is weak in matching the training data is also probable (maybe to a more degree compared to a higher capacity network) to fail on non-seen data.

And when we limit the weights we work around zero, exactly in the linear region of sigmoid function. All of the claims about power of neural network comes from the activation function being nonlinear. If we are working on the linear region, where is that benefit?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I see two points of confusion. First, to predict that poor training recovery means poor prediction assumes that the training set is a representative sample of the entire data space. In high dimensional data, this is often not the case. So relaxing fit to the training data tends improve fit to examples not included in the training. I meant representative of data space, not of the samples. Consider the case where you training set consists primarily of common cases, a few uncommon ones, and several missing cases. Then your model will only perform well on that small portion of the data space; it is over-fit. This is a result of the optimization process, which tries to match the exact data as closely as possible. Between weight sets that give similar predictions for the data, optimization will always favor weights that match the data best, even while others may give better prediction on new data. If your data set is very large and/or highly redundant, then regularization may not improve model accuracy and/or may not shrink the weights very much.

Second, while regularization does move all weights towards zero, it does so in competition with model accuracy. Therefore, at any point in the network, important features will not have their weights shrunk as much as unimportant ones. As long as some weights are still bounded away from zero, then the total activation will still be highly non-linear.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Many thanks for reply. The training sets are basically assumed to be kind of representative. If not why we do the training at all? And if as you said, we "relax fit to training data" why it should lead to "improving fitting to unseen data"? I cannot find a logical relation between these two things. And so do you agree that if the training dataset is representative, regularization will be harmful? $\endgroup$
    – Shahriar49
    Apr 8, 2018 at 17:37
  • $\begingroup$ I think I understand your second paragraph and it means that magically by the nature of neural networks it can determine which weights to keep high while shrinking others to zero. This will be a nonlinear relation if we use sigmoid, which will saturate. But then comes my next confusion. Saturation of sigmoid (which is exactly the reason for nonlinearity) is highly condemned and ReLU is advised instead, which is completely linear in positive portion. Doesn't it matter if all positive activations are exactly linear? Just the negative activations are bad and should be removed? It is confusing. $\endgroup$
    – Shahriar49
    Apr 8, 2018 at 17:39
  • $\begingroup$ I added to my answer to reflect your first question. Saturation is less a product of the non-linearity of the sigmoid but it's horizontal assumptotes. While ReLU lacks asymptotes, it is still non-linear and that non-linearity allows the network to learn non-linear functions. The lack of saturation and the simplicity of the function make it a great choice for deep networks, but sigmoidal functions are still useful when bounded activations are required, such as in LSTM NNs that can be used for natural language processing. $\endgroup$
    – deasmhumnha
    Apr 9, 2018 at 8:45
  • $\begingroup$ Thanks again. For the first point, I understand that optimization tries to match the model to training data, but it is still unanswered that why relaxing this match will give a better match to unseen data. Unseen data has not been seen, we and our model do not know anything about them. How can we expect that all of the sudden magically the network perform better on all unknown things? It may be the case randomly, but we can not say it in general. And if training data are misleading, why to use them at all? A model based on nothing may generalize (by chance) very well. $\endgroup$
    – Shahriar49
    Apr 10, 2018 at 13:25
  • $\begingroup$ And for ReLU, it is actually nonlinear just at one point, zero. I think of nonlinearity as something like saturation or exponential, not two straight lines that just break apart at one point. It is like we are doing nothing (passing without change) for all positive and zeroing for negative values. It is just filtering negative values. Is this really a nonlinear curve compared to sigmoid? And could you please tell me more how we may not need bounded activations? If unbounded, adding layers may result in exploding and saturation is a very logical and natural choice everywhere. $\endgroup$
    – Shahriar49
    Apr 10, 2018 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.