2
$\begingroup$

I just read MAP estimate of linear regression , and got to know that the regularization term is the result of considering prior distribution of weights .

So , my question is how is this prior distribution selected ? Why do we generally use l2/l1 regularization ? Do weights generally have normal/Laplace distribution ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Priors are selected the way you want to penalize your weights. It's not because weights are in general normally or laplacian distributed. L1 regularisation promotes sparsity, while L2 regularisation promotes small weights and go nuts if the absolute value of a weight is high. So, it depends on your purpose. One might use $L_\infty$ norm to limit the maximum of the weights, by assuming a prior of suitable form.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Aren't we deciding the prior distribution of weights on the basis of the true estimate , so that the model doesn't over fit and gives some reliable estimates ? I am not able to connect the dots between regularization and bias variance trade-off from the above explaination . Can you please explain ? $\endgroup$
    – Ajinkya Dandvate
    Commented Sep 20, 2020 at 13:13
  • 2
    $\begingroup$ You should choose your prior "before" you see any data. A MAP estimate essentially assigns the parameters "in between" the prior and the MLE, as you get more data the paramaters move closer to the MLE parameters. Suppose you assign a prior on your weights of a gaussian with very low variance. Then, the data has relatively little influence on your weights (you'd need lots of data to push the weights away from the mean), which shows that the bias is relatively high while the variance should be relatively low. Likewise, requiring so much data to end up with large weights is regularization the sam $\endgroup$
    – harwiltz
    Commented Sep 20, 2020 at 13:49
  • 1
    $\begingroup$ @AjinkyaDandvate I think the comment below yours explains the connection between reg. and bias-variance trade-off very well. On how to select prior, as I mentioned, it's chosen as the way we want to penalize the weights. This is our prior belief on the true weights (not their estimates) before seeing any data. For example, if there are many features, and we believe that the dependent variable is mostly affected by a small subset of them and those weights don't have that much of a problem of being a bit large, we use L1. $\endgroup$
    – gunes
    Commented Sep 20, 2020 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.