I understand that using regularization is equivalent to finding a MAP estimate. I am wondering why using a Gaussian prior (for example) is better at preventing overfitting than using the uniform prior. I can somewhat follow the maths but I cannot form an intuitive reasoning for why this happens.
$\begingroup$
$\endgroup$
4
-
3$\begingroup$ You mean an uniform prior with infinite support (aka not doing regularization, or MLE)? Because if it's an uniform prior with finite support, then it's not so obvious that the Gaussian will tend to perform better. $\endgroup$– FirebugCommented Apr 2, 2021 at 21:32
-
2$\begingroup$ Adding to that, a uniform prior with infinite support would make the MAP parameters the same as the MLE parameters, so you’ve netted yourself nothing in the end. A finite support will mean that you forbid parameter values outside of the support. If you’re optimizing with gradient-based methods, this can put you in hot water quickly. $\endgroup$– Arya McCarthyCommented Apr 2, 2021 at 21:34
-
2$\begingroup$ I mean MLE. I am trying to understand why using some priors such as Gaussian or Laplace is better than using no prior at all. Just the intuition for it $\endgroup$– IiiiCommented Apr 2, 2021 at 21:42
-
1$\begingroup$ It may help to read about shrinkage. $\endgroup$– Arya McCarthyCommented Apr 2, 2021 at 22:31
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Short answer: the improper uniform prior does not put more probability density close to $0$ and thus does not introduce regularization, while the Gaussian and the Laplace prior do that. Moreover, intuitively, the improper uniform prior does put more density mass in any point of the support of the parameter, making the MLE equal to the MAP.
Long answer and comment: I don't think it is always better to use regularization with respect to classical MLE estimate. For simplicity, let's focus on linear regression with gaussian errors.
A clear advantage of regularization is when you have the number of parameters $p$ larger than the number of observations $n$. In such a case, due to the identifiability issue, the classical MLE is not well defined, while regularized versions can be computed (e.g. Lasso, Ridge regression).
The intuition is that the regularization introduces a hard constraint that makes the solution to the optimization problem unique.
Such a solution is equivalent to the MAP in the Bayesian paradigm (e.g. under Laplace, Gaussian prior). The intuition is that the prior does not put a hard constraint, but a soft constraint, assigning different probability densities to the different values of the parameter. Thanks to the soft constraint induced by the prior term the MAP is unique.
Regularization can be used to prevent overfitting also in the case of $p$ smaller than $n$. The idea is that putting some parameters equal to zero, or shrinking them to zero, prevents too complex/flexible models. This can produce a better joint estimate of the parameters in terms of MSE thanks to the fact that even if regularization introduces a bias it can allow to reduce the variance of the estimator. See Stein's paradox for an interesting example.
The same can be done under the Bayesian paradigm, where the posterior distribution (and so the MAP) is a compromise between your prior opinion and the likelihood (Bayes theorem).
