10
$\begingroup$

Or what conditions guarantee that? In general (and not only normal and binomial models) I suppose the main reason that broke this claim is that there's inconsistency between the sampling model and the prior, but what else? I'm starting with this topic, so I really appreciate easy examples

$\endgroup$
9
$\begingroup$

Since the posterior and prior variances on $\theta$ satisfy (with $X$ denoting the sample) $$\text{var}(\theta) = \mathbb{E}[\text{var}(\theta|X)]+\text{var}(\mathbb{E}[\theta|X])$$ assuming all quantities exist, you can expect the posterior variance to be smaller on average (in $X$). This is in particular the case when the posterior variance is constant in $X$. But, as shown by the other answer, there may be realisations of the posterior variance that are larger, since the result only holds in expectation.

To quote from Andrew Gelman,

We consider this in chapter 2 in Bayesian Data Analysis, I think in a couple of the homework problems. The short answer is that, in expectation, the posterior variance decreases as you get more information, but, depending on the model, in particular cases the variance can increase. For some models such as the normal and binomial, the posterior variance can only decrease. But consider the t model with low degrees of freedom (which can be interpreted as a mixture of normals with common mean and different variances). if you observe an extreme value, that’s evidence that the variance is high, and indeed your posterior variance can go up.

$\endgroup$
3
  • $\begingroup$ @Xian, could you take a look at my "answer", which seems to contradict yours? If Gelman and you say something about Bayesian statistics, I am much more inclined to trust you than myself... $\endgroup$ – Christoph Hanck Feb 5 '17 at 7:40
  • 1
    $\begingroup$ No conflict between our answers. There is even an exercise in BDA that corresponds to your example, i.e., find data that sets the Beta posterior variance to be larger than the prior variance. $\endgroup$ – Xi'an Feb 5 '17 at 8:24
  • $\begingroup$ An interesting follow-up question would be: what are the conditions that guarantee convergence of the variance to 0 as sample size increases. $\endgroup$ – Julien Aug 27 '19 at 18:38
8
$\begingroup$

This is going to be more of a question to @Xi'an than an answer.

I was going to answer that a posterior variance $$ V(\theta|y)=\frac{\alpha_{1}\beta_1}{(\alpha_{1}+\beta_1)^2(\alpha_{1}+\beta_1+1)}=\frac{(\alpha _{0}+k)(n-k+\beta_{0})}{(\alpha_{0}+n+\beta_0)^2(\alpha_{0}+n+\beta_0+1)}, $$ with $n$ the number of trials, $k$ the number of successes and $\alpha_{0},\beta_0$ the coefficients of the beta prior, exceeding the prior variance $$ V(\theta)=\frac{\alpha_{0}\beta_0}{(\alpha_{0}+\beta_0)^2(\alpha_{0}+\beta_0+1)} $$ is possible also in the binomial model based on the example below, in which likelihood and prior are in stark contrast so that the posterior is "too far in between". It does seem to contradict the quote by Gelman.

n <- 10         
k <- 1
alpha0 <- 100
beta0 <- 20

theta <- seq(0.01,0.99,by=0.005)
likelihood <- theta^k*(1-theta)^(n-k) 
prior <- function(theta,alpha0,beta0) return(dbeta(theta,alpha0,beta0))
posterior <- dbeta(theta,alpha0+k,beta0+n-k)

plot(theta,likelihood,type="l",ylab="density",col="lightblue",lwd=2)

likelihood_scaled <- dbeta(theta,k+1,n-k+1)
plot(theta,likelihood_scaled,type="l",ylim=c(0,max(c(likelihood_scaled,posterior,prior(theta,alpha0,beta0)))),ylab="density",col="lightblue",lwd=2)
lines(theta,prior(theta,alpha0,beta0),lty=2,col="gold",lwd=2)
lines(theta,posterior,lty=3,col="darkgreen",lwd=2)
legend("top",c("Likelihood","Prior","Posterior"),lty=c(1,2,3),lwd=2,col=c("lightblue","gold","darkgreen"))

 > (postvariance <- (alpha0+k)*(n-k+beta0)/((alpha0+n+beta0)^2*(alpha0+n+beta0+1)))
[1] 0.001323005
> (priorvariance <- (alpha0*beta0)/((alpha0+beta0)^2*(alpha0+beta0+1)))
[1] 0.001147842

Hence, this example suggests a larger posterior variance in the binomial model.

Of course, this is not the expected posterior variance. Is that where the discrepancy lies?

The corresponding figure is

enter image description here

$\endgroup$
2
  • 4
    $\begingroup$ Perfect illustration. And there is no discrepancy between the facts that the realised posterior variance is larger than the prior variance and that the expectation is smaller. $\endgroup$ – Xi'an Feb 5 '17 at 8:22
  • 1
    $\begingroup$ I provided a link to this answer as an excellent example of what was also being discussed here.This result (that variance sometimes increases as data is collected) extends to entropy. $\endgroup$ – Don Slowik Jun 24 '19 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.