It is fairly well known that $$H(X|Y)\le H(X),$$ the posterior entropy is smaller than the prior entropy. This is similar to $$\mathbb{E}_Y[\mathbb{V}ar_X[X|Y]]\le \mathbb{V}ar_X[X]$$ which follows from $\mathbb{V}ar_X[X] = \mathbb{E}_Y[\mathbb{V}ar_X[X|Y]] + \mathbb{V}ar_Y[\mathbb{E}_X[X|Y]]$. In fact, the conditional entropy, $H(X|Y)$, is defined in terms of an average over $Y$, $H(X|Y)=\mathbb{E}_Y[\mathbb{E}_X[\ln(p(X|Y))]]$, so they are quite similar. While these posterior averages are always less than or equal to their respective prior values, for some values of $Y$ we can have $$\mathbb{V}ar_X[X|Y=y]\gt\mathbb{V}ar_X[X], \ \ \ \ \text{and/or}\ \ \ \ \ \mathbb{E}_X[\ln(p(X|Y=y))] \gt H(X).$$ A good example of this is given in this answer. Following that example, a sequence of experiments/data $Y$ can be imagined that leads to the entropy first increasing, then finally decreasing, as illustrated in this other answer. In a Bayesian context, we can have a sequence of conjugate prior distributions for the parameters, $X$, evolving into their updated posteriors recursively via the likelihood given by the data $Y$.

My question is, what can be said about the general situations in which the posterior entropy exceeds the prior? In a sequence of experiments as above where the entropy of the parameter distribution first increases than finally decreases, is there a physical meaning we can ascribe to the regions of increase and of decrease? I am guessing that a 'poor' choice of prior leads to an initial increase in entropy, before the more correct distribution is learned by repeated experiments. Perhaps increasing entropy corresponds to confusion and decreasing to learning?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.