2
$\begingroup$

I was reading a paper and came across this sentence:

"ML methods perform well by employing regularization to reduce variance and trading off regularization bias with overfitting in practice.:

I am wondering if anyone would have any insight into this sentence. I understand that regularization involves shrinkage of parameters, but I fail to see how shrinkage reduces variance and what it means to trade off the bias? Does anyone have any further insights here? Thanks.

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
1
$\begingroup$

The variance referred to here means the variance between different models. Like eg if you fit 5 different nets, from scratch, with different initialization weights, you'll get 5 sets of results on the final test data, or on the dev/validation data. The stronger the regularization, the more similar the results will be, at the expense of fitting the training data less well.

For example, if you use L2 regularization, at the extreme of infinite L2 regularization, the weights will be 0, the output will be 0 for all inputs, and the variance will be zero.

Improve this answer
$\endgroup$
5
  • $\begingroup$ What do you mean by infinite L2 regularization? Is it when, say in a Lasso, $\lambda \to \infty$? $\endgroup$
    – user321627
    Commented Feb 17, 2018 at 19:48
  • $\begingroup$ Like, if you added $\infty \sqrt{ \sum_i w_i^2}$ to your loss function (which would cause your weights to drop immediately to zero) $\endgroup$
    – Hugh Perkins
    Commented Feb 17, 2018 at 19:51
  • $\begingroup$ What is the $\infty$ symbol in front mean? $\endgroup$
    – user321627
    Commented Feb 17, 2018 at 19:52
  • $\begingroup$ So, you would never use infinite L2 regularization. But yes, I mean something like $\lim_{\lambda \rightarrow \infty} \lambda \sqrt{\sum_i w_i^2}$ $\endgroup$
    – Hugh Perkins
    Commented Feb 17, 2018 at 19:53
  • $\begingroup$ (but note that Lasso is L1 regularization; L2 is termed 'ridge', (I think)) $\endgroup$
    – Hugh Perkins
    Commented Feb 17, 2018 at 19:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.