1
$\begingroup$

It is fairly well known that $$H(X|Y)\le H(X),$$ the posterior entropy is smaller than the prior entropy. This is similar to $$\mathbb{E}_Y[\mathbb{V}ar_X[X|Y]]\le \mathbb{V}ar_X[X]$$ which follows from $\mathbb{V}ar_X[X] = \mathbb{E}_Y[\mathbb{V}ar_X[X|Y]] + \mathbb{V}ar_Y[\mathbb{E}_X[X|Y]]$. In fact, the conditional entropy, $H(X|Y)$, is defined in terms of an average over $Y$, $H(X|Y)=\mathbb{E}_Y[\mathbb{E}_X[\ln(p(X|Y))]]$, so they are quite similar. While these posterior averages are always less than or equal to their respective prior values, for some values of $Y$ we can have $$\mathbb{V}ar_X[X|Y=y]\gt\mathbb{V}ar_X[X], \ \ \ \ \text{and/or}\ \ \ \ \ \mathbb{E}_X[\ln(p(X|Y=y))] \gt H(X).$$ A good example of this is given in this answer. Following that example, a sequence of experiments/data $Y$ can be imagined that leads to the entropy first increasing, then finally decreasing, as illustrated in this other answer. In a Bayesian context, we can have a sequence of conjugate prior distributions for the parameters, $X$, evolving into their updated posteriors recursively via the likelihood given by the data $Y$.

My question is, what can be said about the general situations in which the posterior entropy exceeds the prior? In a sequence of experiments as above where the entropy of the parameter distribution first increases than finally decreases, is there a physical meaning we can ascribe to the regions of increase and of decrease? I am guessing that a 'poor' choice of prior leads to an initial increase in entropy, before the more correct distribution is learned by repeated experiments. Perhaps increasing entropy corresponds to confusion and decreasing to learning?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.