# Is adding log-likelihood penalty into optimization problem equivalent to adding prior on model parameters? (why not - in the question)

If we search for a MAP of some $p(\theta \vert D )$ that would look like

$\theta^{MAP} = \arg \max_{\theta} \frac{p(D \vert \theta ) p(\theta)}{Z} = \arg \max_{\theta} p(D \vert \theta ) p(\theta) = \arg \max_{\theta} \log p(D \vert \theta ) p(\theta) = \arg \max_{\theta} \log p(D \vert \theta ) + \log p(\theta)$

in case we add normal prior on $\theta$, that looks like $\theta^{MAP} = \arg \max_{\theta} \log p(D \vert \theta ) - \frac{1}{\sigma}\|\theta\|$ which is pretty much like "fitness + L-* penalty". As soon as I have added the normal prior on params, I expect my params to be just like the prior I just set. But if I optimize that thing with log-likelihood always = constant (no preferences about params) and $\sigma \neq 0$, I would have all components of $\theta = 0$ which is not distributed as a $\mathcal N(0, \sigma)$.

Is it because I have a point estimate of $\theta$? (the "most probable $\theta$ is of course normal). Or is it because my distribution (located around zero) is (in fact) close to normal in terms of Kullback–Leibler divergence (because that zero-distribution is "inside" normal)? Can I somehow "force" the distribution of the values to be close to normal?

UPDATE I figured things out. I actually have each component of the vector being most likelihood according to the normal distribution and that's exactly what I searching for. Sorry for the mess.

• You said if the likelihood term (I assume you mean the first) and you optimize you'll get $\theta =0$. But for the first term to be constant (with respect to theta) that means your data does not depend on $\theta$. Why would you be doing this? If I told you I have a normally distributed random variable with mean 0 and asked you to estimate it, you would say 0. That doesn't mean it's not normally distributed.
• I think it's important to point out that adding a prior and then taking the MAP solution as a point estimate is not the only way (or even the normal way) in which you would do Bayesian parameter estimation. Rather, you'd normally treat the entire posterior distribution as the answer rather than any single point estimate. There's a world of difference between "The MLE is $\hat{\theta}=27$" and "The posterior distribution for $\theta$ is shown in this histogram." – Brian Borchers Sep 11 '15 at 3:19