Posterior variance reduction

Question

As detailed on its Wikipedia page, Mutual information, $I(X,Y)$, can be bounded by the Jensen inequality to show that it is always positive. Also, one can show that $$ I(X,Y) = H(X) - H(X|Y). $$ Together this implies that $$H(X|Y) < H(X). $$ If seen through Bayes Rule, this implies that information on $X$ usually increases given the observation $Y$ (for any family of distributions).

Now, if $X \sim N(0,P)$ and $Y = X + \xi$, and $\xi \sim N(0,R)$, then $X|y \sim N\left(0,(1/R + 1/P)^{-1}\right)$. Similar results apply with non-zero means, linear functions on $X$ or $\xi$, and in the multivariate case. The thing to note is that the posterior variance is guaranteed to be smaller than the prior variance

Since variance can in some way be seen as a measure on the information we know of a variable (e.g. the Fisher information) my question is: are there any conditions (other than from the example above) under which the posterior variance is guaranteed to be smaller than the prior variance? This would be analogous to the above result on entropy.

I suppose one could start looking at other conjugate distributions than the Gaussian, but that seems too specific.

Other comments on the viability of this quest, or just an interpretation of somethings, are very welcome.

Answer 1

As you point out, $$H(X|Y)\le H(X)$$ is generally true, and can be interpreted from a Bayesian perspective as the entropy decrease in $X$ going from the prior to posterior distributions upon incorporating the additional information provided by observation of the data $Y$.

It is also generally true that $\mathbb{V}ar_X[X] = \mathbb{E}_Y[\mathbb{V}ar_X[X|Y]] + \mathbb{V}ar_Y[\mathbb{E}_X[X|Y]]$ and therefore $$\mathbb{E}_Y[\mathbb{V}ar_X[X|Y]]\le \mathbb{V}ar_X[X]$$ i.e., the mean variance in the posterior, $X|Y$, is less than that of the prior, $X$.

These reductions in both entropy and variance in going from the prior to posterior distributions of $X$ are statements about expectations over $Y$. Recall that conditional entropy is defined as $H(X|Y)=\mathbb{E}_Y[\mathbb{E}_X[\ln(p(X|Y))]]$, so it really is an average over $Y$ of the entropy of the posterior.

Since these statements are about expectations, they leave open the possibilities that, for some $y$, we could have $$\mathbb{V}ar_X[X|Y=y]\gt\mathbb{V}ar_X[X], \ \ \ \ \text{and/or}\ \ \ \ \ \mathbb{E}_X[\ln(p(X|Y=y))] \gt H(X).$$

There is an excellent example of this phenomenon provided by this answer . Taking that example a step further. I calculated both the entropy and the variance of the posterior (conditional) distribution using the numbers and the beta / binomial set-up of that example: In R:

> a0 <- 100
> b0 <- 20
> a <- a0 + 1
> b <- b0 + 9
> (postvar <- a*b / ((a+b)^2 * (a+b+1)))
[1] 0.001323005
> (priorvar <- a0*b0 / ((a0+b0)^2 * (a0+b0+1)))
[1] 0.001147842
> (postentropy <- log(beta(a,b)) - (a-1)*digamma(a) - (b-1)*digamma(b) + (a+b-2)*digamma(a+b)
[1] -1.899637
> (prie=log(beta(a0,b0))-(a0-1)*digamma(a0)-(b0-1)*digamma(b0)+(a0+b0-2)*digamma(a0+b0))
[1] -1.97511

So we see that what they found there: that the variance of the parameters' beta distribution increased after the data were collected, holds true of the entropy as well. I used the formula for entropy from here. Now continuing in python / scipy (I reproduced the above to make contact with that variance example.)

In [1]: from scipy.stats import beta

In [2]: a0, b0 = 100, 20

In [3]: a, b = 100+1, 20+9

In [4]: beta.var(a0, b0) # Prior variance
Out[4]: 0.001147842056932966

In [5]: beta.var(a, b)   # Posterior variance
Out[5]: 0.0013230046524233252

In [6]: beta.entropy(a0, b0)  # Prior entropy
Out[6]: array(-1.97510984)

In [7]: beta.entropy(a, b)    # Posterior entropy
Out[7]: array(-1.89963714)

In [8]: beta.entropy(1, 1)  # uniform entropy
Out[8]: array(0.)

In [9]: beta.entropy(1000, 1000)  # sharply peaked beta
Out[9]: array(-3.07491)

In [10]: beta.entropy(100000, 100000)  # sharply peaked beta
Out[10]: array(-5.37724747)

In [11]: a, b = 100, 20

In [12]: for i in range(14):
    ...:     print(a, b, beta.var(a,b), beta.entropy(a,b))
    ...:     a, b = a+1, b+9
    ...:     
100 20 0.001147842056932966 -1.9751098394063353
101 29 0.0013230046524233252 -1.8996371404250594
102 38 0.0014025184541901867 -1.868390558158729
103 47 0.0014248712288447386 -1.8593872860958125
104 56 0.0014130434782608694 -1.8629279074120686
105 65 0.0013810477751472106 -1.874007722570822
106 74 0.0013375622399563467 -1.889781703473504
107 83 0.0012880161273948166 -1.9085230031335483
108 92 0.001235820895522388 -1.929133400999706
109 101 0.0011831146360597952 -1.9508901652539663
110 110 0.0011312217194570135 -1.9733054939285433
111 119 0.0010809417425674515 -1.9960440293558486
112 128 0.00103273397879207 -2.0188722674726316
113 137 0.0009868366533864541 -2.0416263912578745

So we find that if we keep getting the same result (1 success in 10 trials) 14 times, the variance and entropy of the beta distribution for the parameter first increases, then begins to decrease from then on as the beta distribution becomes better defined by the data.