21
$\begingroup$

If the prior and the likelihood are very different from each other, then sometimes a situation occurs where the posterior is similar to neither of them. See for example this picture, which uses normal distributions.

Posterior Behaviour

Although this is mathematically correct, it doesn't seem to accord with my intuition -- if the data doesn't match with my strongly held beliefs or the data, I would expect neither range to fare well and would expect either a flat posterior over the whole range or perhaps a bimodal distribution around the prior and likelihood (I'm not sure which makes more logical sense). I certainly wouldn't expect a tight posterior around a range that matches neither my prior beliefs or the data. I understand that as more data is collected, the posterior will move towards the likelihood, but in this situation it seems counter-intuitive.

My question is: how is my understanding of this situation flawed (or is it flawed). Is the posterior the `correct' function for this situation. And if not, how else might it be modelled?

For completeness sake, the prior is given as $\mathcal{N}(\mu=1.5, \sigma=0.4)$ and the likelihood as $\mathcal{N}(\mu=6.1, \sigma=0.4)$.

EDIT: Looking at some of the answers given, I feel like I haven't explained the situation very well. My point was the the Bayesian analysis seems to produce a non-intuitive result given the assumptions in the model. My hope was that the posterior would somehow `account' for perhaps bad modelling decisions, which when thought about is definitely not the case. I'll expand upon this in my answer.

$\endgroup$
  • 2
    $\begingroup$ That would then simply mean that you cannot assume normality of the posterior. If you assume the posterior is normal, then this would indeed be correct. $\endgroup$ – PascalVKooten Feb 13 '14 at 17:25
  • $\begingroup$ I didn't make any assumption on the posterior, only the prior and likelihood. And in any case, the form of the distribution seems irrelevant here - I could have drawn them manually and the same posterior would follow. $\endgroup$ – Rónán Daly Feb 13 '14 at 17:35
  • $\begingroup$ I'm just saying that you would throw away your belief in this posterior if you don't assume the posterior can be normal. Considering a normal prior and normal data, a normal posterior would indeed be like this. Perhaps imagine small data, something like this might then actually occur in reality. $\endgroup$ – PascalVKooten Feb 13 '14 at 17:56
  • 1
    $\begingroup$ Is this figure correct? It seems that the likelihood $\times$ prior should be very close to 0 as they never overlap. I am having trouble seeing how your posterior can peek there as the weight of the prior is very close to 0 there. Am I missing something? $\endgroup$ – Luca Feb 13 '14 at 17:59
  • 1
    $\begingroup$ @Luca You're forgetting about re-normalisation. The product of the prior and likelihood is close to zero, yes - but when you re-normalise it so it integrates to 1 again, this becomes irrelevant. $\endgroup$ – Pat Feb 13 '14 at 18:28
5
$\begingroup$

Yes this situation can arise and is a feature of your modeling assumptions specifically normality in the prior and sampling model (likelihood). If instead you had chosen a Cauchy distribution for your prior, the posterior would look much different.

prior = function(x) dcauchy(x, 1.5, 0.4)
like = function(x) dnorm(x,6.1,.4)

# Posterior
propto = function(x) prior(x)*like(x)
d = integrate(propto, -Inf, Inf)
post = function(x) propto(x)/d$value

# Plot
par(mar=c(0,0,0,0)+.1, lwd=2)
curve(like, 0, 8, col="red", axes=F, frame=T)
curve(prior, add=TRUE, col="blue")
curve(post, add=TRUE, col="seagreen")
legend("bottomleft", c("Prior","Likelihood","Posterior"), col=c("blue","red","seagreen"), lty=1, bg="white")

Cauchy prior, normal sampling model

$\endgroup$
  • $\begingroup$ Thanks for your answer @jaradniemi, would you think a Cauchy prior would always avoid the particular situation given in the question? $\endgroup$ – Rónán Daly Feb 19 '14 at 10:35
  • 1
    $\begingroup$ Yes. Generally heavy tailed priors allow the data to more easily overwhelm the prior. $\endgroup$ – jaradniemi Mar 5 '14 at 16:28
  • 2
    $\begingroup$ jaradniemi, that may be so, but if you say you don't want your prior to influence the posterior, why do you choose an informative prior in the first place? It seems like you are suggesting to choose a cauchy because it looks informative, but it actually isn't. $\endgroup$ – Florian Hartig Oct 11 '15 at 11:14
  • 1
    $\begingroup$ If the prior and likelihood agree, then you get the desired increase in accuracy from the prior to the posterior and thus the prior is informative. But choosing a heavy-tailed prior allows for the likelihood to easily overwhelm the prior when the two disagree. $\endgroup$ – jaradniemi Feb 9 '16 at 19:24
2
$\begingroup$

I somewhat disagree with the answers given so far - there is nothing weird about this situation. The likelihood is asymptotically normal anyway, and a normal prior is not uncommon at all. If you put both together, with the fact that prior and likelihood don't give the same answer, we have the situation we are talking about here. I have depicted that below with the code by jaradniemi.

We mention in 1 that the normal conclusion of such an observation would be that either a) model is structurally wrong b) data is wrong c) prior is wrong. But something is wrong for sure, and you would also see this if you'd do some posterior-predictive checks, which you should do anyway.

1 Hartig, F.; Dyke, J.; Hickler, T.; Higgins, S. I.; O'Hara, R. B.; Scheiter, S. & Huth, A. (2012) Connecting dynamic vegetation models to data - an inverse perspective. J. Biogeogr., 39, 2240-2252. http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2699.2012.02745.x/abstract

prior = function(x) dnorm(x,1,.3)
like = function(x) dnorm(x,-1,.3)

# Posterior
propto = function(x) prior(x)*like(x)
d = integrate(propto, -Inf, Inf)
post = function(x) propto(x)/d$value

# Plot
par(mar=c(0,0,0,0)+.1, lwd=2)
curve(like, -2, 2, col="red", axes=F, frame=T, ylim = c(0,2))
curve(prior, add=TRUE, col="blue")
curve(post, add=TRUE, col="seagreen")
legend("bottomleft", c("Prior","Likelihood","Posterior"), col=c("blue","red","seagreen"), lty=1, bg="white")

enter image description here

$\endgroup$
2
$\begingroup$

I feel like the answer that I was looking for when it came to this question is best summarized by Lesaffre and Lawson in Bayesian Biostatistics

The posterior precision is the sum of the prior and the sample precision, i.e.: $$ \frac{1}{\sigma^2} = w_{0} + w_{1} $$ This shows that the posterior is more peaked than the prior and the likelihood function which means that the posterior contains more information about $\mu$ than the prior and the likelihood function. This property holds even when the likelihood and prior are in conflict (in contrast to the binomial-beta case). This may look counterintuitive since, in the presence of conflicting information, there is more uncertainty a posteriori rather than less uncertainty. Note that this result only holds for the special, and unrealistic, case of a known $\sigma$.

What this summarizes for me, and is roughly outlined in the other answers, is that the case of modeling normal priors with a normal likelihood can result in a situation where the posterior is more precise than either. This is counterintuitive, but is a special consequence of modeling these elements this way.

$\endgroup$
  • $\begingroup$ This generalizes in higher dimension with the Fisher matrix. The Hessian of the log likelihood of the posterior distribution near its peak is the sum of the prior and likelihood inverse covariances. The inverse of this sum is the covariance of the posterior. Because two positive (semi) definite matrices (inverse covariances) are added, it is mathematically guaranteed that the precision of the posterior will exceed that of either the prior or the likelihood probability distributions. This is a universal result in the Bayesian framework. $\endgroup$ – T3am5hark Feb 17 '17 at 15:58
2
$\begingroup$

If your model is correct, observing a likelihood function falling this far from the prior is extremely unlikely. Imagine the situation before seeing the data $X_1$. Based on on correct inferences from past data $X_0$ you believe that $\mu \sim N(1.6, 0.4^2)$. Suppose that the future set of data is a single normal variate $X_1 \sim N(\mu, 0.4^2)$. The likelihood will then have its maximum at $X_1$. From this it follows that the location of the likelihood functions maximum, once $X_1$ is observed, will fall at a normally distributed position with expectation equal to your prior mean 1.6 and standard deviation equal to $\sqrt{0.4^2 + 0.4^2}=0.56$. So the probability that it falls at 6.1 (as in your example) or even further away from your prior mean is $2\phi(-(6.1-1.6)/0.56)=9.3\cdot 10^{-16}$. So this is not going to happen in practice provided that your model including past inferences about $\mu$ are correct.

For simplicity, suppose that the "past data" was also a single variate $X_0 \sim N(\mu,0.4^2)$ and that you started off with a flat prior before observing $X_0$. Put differently, in terms of $X_0$ and $X_1$ the above situation or something more extreme arise if $|X_1-X_0|>6.1-1.6$ which is extremely unlikely given that both variates according to the model should have come from the same normal distribution with a standard deviation of 0.4.

So if you come across something like this in practice, it would strongly suggest that either the inference made from $X_0$ or $X_1$ (or both) are wrong and in some way have ignored important sources of uncertainty. If you believed this to be the case but you happened to be unable to find any error, I would say the best you could do would be to adjust the standard deviations of both likelihood and prior post hoc to make them more compatible. This would indeed result in an adjusted wider posterior more compatible with your intuition.

$\endgroup$
1
$\begingroup$

After thinking about this for a while, my conclusion is that with bad modelling assumptions, the posterior can be a result that accords with neither prior beliefs or the likelihood. From this the natural result is the the posterior is not, in general, the end of the analysis. If it is the case that the posterior should roughly fit the data or that it should be diffuse between the prior and likelihood (in this case), then this would have to be checked after the fact, probably with a posterior-predictive check or something similar. To incorporate this into the model would seem to require the ability to put probabilities on probabilistic statements, which I don't think is possible.

$\endgroup$
  • $\begingroup$ yes, I agree, see my more detailed answer $\endgroup$ – Florian Hartig Oct 11 '15 at 11:10
0
$\begingroup$

I think this is actually a really interesting question. Having slept on it, I think I have a stab at an answer. The key issue is as follows:

  • You've treated the likelihood as a gaussian pdf. But it's not a probability distribution - it's a likelihood! What's more, you've not labelled your axis clearly. These things combined have confused everything which follows.

Let's say you're inferring the mean of a normal distribution, $\mu$. It's a one dimensional plot, so I'll assume $\sigma$ is known. In that case your prior distribution must be $P(\mu|\mu', \sigma')$, where $\mu'$ and $\sigma'$ are (fixed) hyperparameters controlling your prior's position and shape; your likelihood function is $P(X|\mu, \sigma)$, where $X$ is your observed data; and your posterior is $P(\mu|X, \sigma, \mu', \sigma')$. Given that, the only horizontal axis that makes sense to me in this diagram is one which is plotting $\mu$.

But if the horizontal axis shows values of $\mu$, why does the likelihood $P(X|\mu)$ have the same width and height as the prior? When you break it down that's actually a really weird situation. Think about the form the the prior and likelihood:

$$ P(\mu|\mu', \sigma') = exp(-\frac{(\mu-\mu')^2}{2 \sigma'^2})\frac{1}{\sqrt{2 \pi \sigma'^2}} $$

$$ P(X|\mu,\sigma) = \prod_{i=1}^N exp(-\frac{(x_i-\mu)^2}{2 \sigma^2})\frac{1}{\sqrt{2 \pi \sigma^2}} $$

The only way I can see that these can have the same width is if $\sigma'^2 = \sigma^2/N$. In other words, your prior is very informative, as its variance is going to be much lower than $\sigma^2$ for any reasonable value of $N$. It is literally as informative as the entire observed dataset $X$!

So, the prior and the likelihood are equally informative. Why isn't the posterior bimodal? This is because of your modelling assumptions. You've implicitly assumed a normal distribution in the way this is set up (normal prior, normal likelihood), and that constrains the posterior to give a unimodal answer. That's just a property of normal distributions, that you have baked into the problem by using them. A different model would not necessarily have done this. I have a feeling (though lack a proof right now) that a cauchy distribution can a have multimodal likelihood, and hence a multimodal posterior.

So, we have to be unimodal, and the prior is as informative as the likelihood. Under these constraints, the most sensible estimate is starting to sound like a point directly between the likelihood and prior, as we have no reasonable way to tell which to believe. But why does the posterior get tighter?

I think the confusion here comes from the fact that in this model, $\sigma$ is assumed to be known. Were it unknown, and we had a two dimensional distribution over $\mu$ and $\sigma$ the observation of data far from the prior might make a high value of $\sigma$ more probable, and so increase the variance of the posterior distribution of the mean too (as these two are linked). But we're not in that situation. $\sigma$ is treated as known here. A such adding more data can only make us more confident in our prediction of the position of $\mu$, and hence the posterior becomes narrower.

(A way to visualise it might be to imagine estimating the mean of a gaussian, with known variance, using just two sample points. If the two sample points are separated by very much more than the width of the gaussian (i.e. they're out in the tails), then that's strong evidence the mean actually lies between them. Shifting the mean just slightly from this position will cause an exponential drop off in the probability of one sample or another.)

In summary, the situation you have described is a bit odd, and by using the model you have you've included some assumptions (e.g. unimodality) into the problem that you didn't realise you had. But otherwise, the conclusion is correct.

$\endgroup$
  • $\begingroup$ Thanks for your answer @Pat, I agree with most of what you say here, the problem as set up was a bit sloppy (though the likelihood is simply a function of the parameters, so it's fine for it to be a probability density for sake of example). I should really do the analysis for $\mu$ and $\sigma$ free, but do you think a diffuse prior on $\sigma$ would always avoid the situation given in the question for the marginal posterior of $\mu$? $\endgroup$ – Rónán Daly Feb 19 '14 at 10:41

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.