2
$\begingroup$

It's well known that the MLE $\hat{\theta}$ maximizes $f(y\mid\theta)$ and under regularity conditions has asymptotic distribution $$N\left(\theta, \frac{I(\theta)}{J^2(\theta)} \right)$$ where $I(\theta)=Var(E[\partial\ell'(y_i,\theta)])$ and $J(\theta)=-E[\partial^2\ell(y_i,\theta)]$(for simplicity I treat the single parameter case).

the MAP estimate gives $\theta$ a prior density and seeks to maximize $f(\theta\mid y)$. It can also be shown that $f(\theta\mid y)$ in the limit converges to $$N(\hat{\theta}, -\partial^2\ell(\theta\mid y))|_{\theta=\hat{\theta}})$$ where $\hat{\theta}$ denotes the mean of the posterior distribution, ie the MAP. Can I use this result to attain the asymptotic distribution of the MAP, similarly to the MLE? I thought it might have something to do with conjugate distributions (about which I know little)? It would already be very helpful if this were true in a special case, ie the normal case.

$\endgroup$

1 Answer 1

5
$\begingroup$

Let $g(\theta)$ be the prior distributionfor $\theta$. Let $\mathcal L_n$ be the log-likelhood of the sample,

$$\mathcal L_n = \sum_{i=1}^n \ln f(y_i \mid \theta)$$

The MLE maximizes just $\mathcal L_n$, or equivalently $(1/n)\mathcal L_n$.

The MAP maximizes (by considering the monotonic logarithmic transformation) $\ln g(\theta) + \mathcal L_n$, so the f.o.c. is

$$\hat \theta_{MAP}: \frac{g'(\hat \theta_{MAP})}{g(\hat \theta_{MAP})}+\frac{\partial \mathcal L_n(\hat \theta_{MAP})}{\partial \theta} = 0.$$

But, then we also have

$$\hat \theta_{MAP}: \frac 1n \left[\frac{g'(\hat \theta_{MAP})}{g(\hat \theta_{MAP})}+\frac{\partial \mathcal L_n(\hat \theta_{MAP})}{\partial \theta} \right]= 0$$

$$\implies \hat \theta_{MAP}: \frac 1n \frac{g'(\hat \theta_{MAP})}{g(\hat \theta_{MAP})}+\frac 1n\frac{\partial \mathcal L_n(\hat \theta_{MAP})}{\partial \theta} = 0.$$

But this implies that as $n \to \infty$ the first term goes to zero, because it is not some sum that increases as sample size increases, and so the MAP will become stochastically equivalent to the MLE. Intuitively, as we increase the size of the sample, the prior information embodied in $g(\theta)$ becomes negligible, compared to the information contained in the ever-increasing sample.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.