# Entropy evolution while learning?

It is fairly well known that $$H(X|Y)\le H(X),$$ the posterior entropy is smaller than the prior entropy. This is similar to $$\mathbb{E}_Y[\mathbb{V}ar_X[X|Y]]\le \mathbb{V}ar_X[X]$$ which follows from $$\mathbb{V}ar_X[X] = \mathbb{E}_Y[\mathbb{V}ar_X[X|Y]] + \mathbb{V}ar_Y[\mathbb{E}_X[X|Y]]$$. In fact, the conditional entropy, $$H(X|Y)$$, is defined in terms of an average over $$Y$$, $$H(X|Y)=\mathbb{E}_Y[\mathbb{E}_X[\ln(p(X|Y))]]$$, so they are quite similar. While these posterior averages are always less than or equal to their respective prior values, for some values of $$Y$$ we can have $$\mathbb{V}ar_X[X|Y=y]\gt\mathbb{V}ar_X[X], \ \ \ \ \text{and/or}\ \ \ \ \ \mathbb{E}_X[\ln(p(X|Y=y))] \gt H(X).$$ A good example of this is given in this answer. Following that example, a sequence of experiments/data $$Y$$ can be imagined that leads to the entropy first increasing, then finally decreasing, as illustrated in this other answer. In a Bayesian context, we can have a sequence of conjugate prior distributions for the parameters, $$X$$, evolving into their updated posteriors recursively via the likelihood given by the data $$Y$$.

My question is, what can be said about the general situations in which the posterior entropy exceeds the prior? In a sequence of experiments as above where the entropy of the parameter distribution first increases than finally decreases, is there a physical meaning we can ascribe to the regions of increase and of decrease? I am guessing that a 'poor' choice of prior leads to an initial increase in entropy, before the more correct distribution is learned by repeated experiments. Perhaps increasing entropy corresponds to confusion and decreasing to learning?