6
$\begingroup$

When is maximum a posterior (MAP) estimation equivalent to maximum-likelihood (ML) estimation?

$\endgroup$
5
$\begingroup$

One can proof that in the limit of infinite data, both estimates converge.

Let us consider the case of regression, where you assume that the target data is generated from a smooth function width additive Gaussian noise. Then you have for the likelihood of your training data, $$p(D|\mathbf{w}) = \prod_{n} p(t_{n}|\mathbf{x_{n}},\mathbf{w}) = \prod_{n}\exp \left(\frac{\beta}{2} \left[t_{n}- y(\mathbf{x_{n}},\mathbf{w}) \right]^{2}\right)/Z_{D}(\beta)$$

where $\mathbf{w}$ is a vector containing all parameters which characterize your algorithm and $Z_{D}(\beta)$ is a normalization constant. If you maximize the log-likelihood if this expression you get the ML estimate.

Now, you add a prior on the parameters which acts as a regularizer and helps you avoid overfitting by controlling the complexity of your classifier. Concretely, in the case it is natural to assume that your parameters are Gaussian distributed,

$$p(\mathbf{w}) = \exp \left( -\frac{\alpha ||\mathbf{w}||^{2}}{2}\right)/Z_{W}(\alpha)$$

MAP is defined as $\arg\max_{w} p(\mathbf{w}|D)$. Using Bayes' theorem,

$$p(\mathbf{w}|D) = p(D|\mathbf{w})p(\mathbf{w})$$ If you substitute the above expressions and take logarithms you end up with (the $Z$'s do not depend on $\mathbf{w}$),

$$\arg\min_{w} \sum_{n}\frac{\beta}{2} \left[t_{n}- y(\mathbf{x_{n}},\mathbf{w}) \right]^{2} + \frac{\alpha}{2}\sum_{i}w_{i}^{2}$$

which is nothing more as ridge regression. The more data you add, the bigger the first term will be in comparison to the second, i.e. the closer to the ML estimate. A very similar derivation can be done for the case of classification.

If you are interested on Machine Learning, I would recommend you to get a copy of Bishop's book.

$\endgroup$
  • $\begingroup$ I think that juampa's answer above is partly correct. If the MLE for a parameter is $\hat{\theta}$ for example and the prior assigns probability $0$ to $\hat{\theta}$, then no matter how large the likelihood becomes, the posterior probability of $\hat{\theta}$ will always be $0$. Therefore, MAP tends to MLE with infinite data only as long as the prior we select assigns a positive (i.e., non-zero) probability to the actual MLE value. Please correct me if I am wrong! $\endgroup$ – user30070 Sep 22 '14 at 3:35
  • $\begingroup$ @juampa why is the (logarithm of) prior $\frac{\alpha}{2}\sum_{i}w_{i}^{2}$? $\endgroup$ – Lerner Zhang Feb 27 '18 at 11:27
  • $\begingroup$ That seems like Tikhonov regularization... $\endgroup$ – Lerner Zhang Feb 27 '18 at 11:34
3
$\begingroup$

At Wikipédia article maximum a posterior and also at slide 17 of this presentation here, you will find that both estimators coincide when you use an uniform prior over the support of the likelihood function (improper "flat" prior if the support is infinite).

EDIT: I'm pretty sure this is valid for location parameters, but not sure and couldn't found if this is also valid for scale parameters. I think that for those, one needs to use Jeffreys' prior, but I'm not sure, I'd be glad if someone answers that.

$\endgroup$
  • $\begingroup$ it seems that your answer conflicts with that of @juampa. You said the prior is a uniform but in his answer it seems not the case. $\endgroup$ – Lerner Zhang Feb 27 '18 at 11:36
  • $\begingroup$ @lerner I don't think they do. There's two ways to think about it: first, under some mild conditions MAP and MLE will be the same asymptotically (with infinite amount of data) for "any" prior. The obviously scenario would be when both estimators are consistent. But what I was saying here in my answer was that for some special cases with a given prior, both MLE and MAP will be the same for any dataset. $\endgroup$ – random_user Feb 27 '18 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.