If we search for a MAP of some $p(\theta \vert D )$ that would look like

$\theta^{MAP} = \arg \max_{\theta} \frac{p(D \vert \theta ) p(\theta)}{Z} = \arg \max_{\theta} p(D \vert \theta ) p(\theta) = \arg \max_{\theta} \log p(D \vert \theta ) p(\theta) = \arg \max_{\theta} \log p(D \vert \theta ) + \log p(\theta)$

in case we add normal prior on $\theta$, that looks like $\theta^{MAP} = \arg \max_{\theta} \log p(D \vert \theta ) - \frac{1}{\sigma}\|\theta\|$ which is pretty much like "fitness + L-* penalty". As soon as I have added the normal prior on params, I expect my params to be just like the prior I just set. But if I optimize that thing with log-likelihood always = constant (no preferences about params) and $\sigma \neq 0$, I would have all components of $\theta = 0$ which is not distributed as a $\mathcal N(0, \sigma)$.

Is it because I have a point estimate of $\theta$? (the "most probable $\theta$ is of course normal). Or is it because my distribution (located around zero) is (in fact) close to normal in terms of Kullback–Leibler divergence (because that zero-distribution is "inside" normal)? Can I somehow "force" the distribution of the values to be close to normal?

UPDATE I figured things out. I actually have each component of the vector being most likelihood according to the normal distribution and that's exactly what I searching for. Sorry for the mess.


This question is a bit all over the map (get it?) but I'll answer what I can.

  • Yes adding a prior and doing a MAP estimate is often (always?) equivalent to adding a penalty term to an ordinary MLE estimate (though in your case a normal prior should lead to a squared norm (and you should have argmax everywhere or a few negative signs).

  • You said if the likelihood term (I assume you mean the first) and you optimize you'll get $\theta =0$. But for the first term to be constant (with respect to theta) that means your data does not depend on $\theta$. Why would you be doing this? If I told you I have a normally distributed random variable with mean 0 and asked you to estimate it, you would say 0. That doesn't mean it's not normally distributed.

If you can smooth out some of the rest of your question I'll try to answer it.

| cite | improve this answer | |
  • $\begingroup$ I think it's important to point out that adding a prior and then taking the MAP solution as a point estimate is not the only way (or even the normal way) in which you would do Bayesian parameter estimation. Rather, you'd normally treat the entire posterior distribution as the answer rather than any single point estimate. There's a world of difference between "The MLE is $\hat{\theta}=27$" and "The posterior distribution for $\theta$ is shown in this histogram." $\endgroup$ – Brian Borchers Sep 11 '15 at 3:19
  • $\begingroup$ Of course. But there tends to come a time in life when you need a number, for instance, how much is this thing worth because we're going to make an offer. At times like that MAP and Bayesian Inference (expected value of posterior) are very important tools. $\endgroup$ – jlimahaverford Sep 11 '15 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.