8
$\begingroup$

It is usually said that priors on bayesian statistics can be regarded as regularization factors since they penalize solutions where the prior places low density of probability.

Then, given this simple model whose MLE parameters are: $$ argmax_{\mu} \text{ } \mathcal{N}(y; \mu, \sigma) $$

and I add a prior: $$ argmax_{\mu} \text{ } \mathcal{N}(y; \mu, \sigma) \mathcal{N}(\mu; 0, \sigma_0) $$ the parameters are not the MLE parameters but the MAP parameters.

Question: Does this mean that if I introduce some regularization in my model I am doing a bayesian analysis (even if only use point-estimates)?

Or this just makes no sense making this "ontological" distinction at this point since the method to find either MLE or MAP is the same (isn't it?)?

$\endgroup$
10
$\begingroup$

It means that the analysis has a Bayesian interpretation, but that doesn't mean that it might not also have a frequentist interpretation as well. The MAP estimate might be viewed as being a partialy Bayesian approach, with a more complete Bayesian approach being to consider the posterior distribution over the parameters. It is still a Bayesian approach though, as the definition of probability would be a "degree of plausibility", rather than a long run frequency.

$\endgroup$
5
$\begingroup$

If you use the L2 norm, i.e., quadratic penalty on the log likelihood function, penalization is very similar to a Bayesian procedure with a Gaussian prior with mean zero for the non-intercept regression coefficients. But unlike the full Bayesian procedure that factors in the uncertainty about the the amount of penalization (analogous to treating the variance of random effects as if it were a known constant), the penalized maximum likelihood procedure pretends that the optimum penalty was pre-specified and is not an unknown parameter. So it results in confidence limits that are a bit too narrow.

$\endgroup$
  • 2
    $\begingroup$ +1 good point, not taking into account the uncertainty in the hyper-parameters is Empirical Bayes. $\endgroup$ – Dikran Marsupial Nov 29 '14 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.