1
$\begingroup$

I was reading a paper and came across this sentence:

"ML methods perform well by employing regularization to reduce variance and trading off regularization bias with overfitting in practice.:

I am wondering if anyone would have any insight into this sentence. I understand that regularization involves shrinkage of parameters, but I fail to see how shrinkage reduces variance and what it means to trade off the bias? Does anyone have any further insights here? Thanks.

$\endgroup$
1
$\begingroup$

The variance referred to here means the variance between different models. Like eg if you fit 5 different nets, from scratch, with different initialization weights, you'll get 5 sets of results on the final test data, or on the dev/validation data. The stronger the regularization, the more similar the results will be, at the expense of fitting the training data less well.

For example, if you use L2 regularization, at the extreme of infinite L2 regularization, the weights will be 0, the output will be 0 for all inputs, and the variance will be zero.

$\endgroup$
  • $\begingroup$ What do you mean by infinite L2 regularization? Is it when, say in a Lasso, $\lambda \to \infty$? $\endgroup$ – user321627 Feb 17 '18 at 19:48
  • $\begingroup$ Like, if you added $\infty \sqrt{ \sum_i w_i^2}$ to your loss function (which would cause your weights to drop immediately to zero) $\endgroup$ – Hugh Perkins Feb 17 '18 at 19:51
  • $\begingroup$ What is the $\infty$ symbol in front mean? $\endgroup$ – user321627 Feb 17 '18 at 19:52
  • $\begingroup$ So, you would never use infinite L2 regularization. But yes, I mean something like $\lim_{\lambda \rightarrow \infty} \lambda \sqrt{\sum_i w_i^2}$ $\endgroup$ – Hugh Perkins Feb 17 '18 at 19:53
  • $\begingroup$ (but note that Lasso is L1 regularization; L2 is termed 'ridge', (I think)) $\endgroup$ – Hugh Perkins Feb 17 '18 at 19:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.