# If $T$ is a sufficient statistic for $\theta$, is $H(\theta\mid x) = H(\theta\mid T(x))$?

I was trying to prove that sufficient statistics attain equality in the data processing inequality by a slightly different route than I usually see, and came across an odd expression. (I care more about understanding the odd expression than about proving the data processing inequality).

So if we have $$\theta \longrightarrow X \longrightarrow T(X)$$, the data processing inequality gives us

$$I(\theta;X) \ge I(\theta;T(X))\tag{1}$$

and expanding out into entropies yields $$H(\theta) - H(\theta\mid X) \ge H(\theta) - H(\theta\mid T(X))$$, which yields $$H(\theta\mid X) \le H(\theta\mid T(X))\tag{2}$$

So $$(1)$$ attains equality iff $$(2)$$ does. But how on earth do I even interpret $$(2)$$? $$T(X)$$ is generally a non-injective mapping, and so it seems very strange to me that a non-injective mapping (which should strictly decrease discrete entropy) could possibly lead to an equality.

I was trying to manipulate the Neyman-Fisher factorization to try and flip into the form of $$(2)$$, but to no avail.

Any thoughts?

Thanks!

Yes, it is true that if $$T$$ is sufficient, then $$H(\theta | x) = H(\theta|T(x))$$.

From the definition of sufficiency, we have that $$P(x|T(x), \theta) = P(x|T(x))$$.

And now

$$P(\theta | x) = \frac{P(x, \theta)}{P(x)} = \frac{P(x, T(x), \theta)}{P(x, T(x))} = \frac{P(x|T(x), \theta)P(\theta, T(x))}{P(x|T(x))P(T(x))}$$

Now we use the above fact to simplify:

$$\frac{P(x|T(x), \theta)P(\theta, T(x))}{P(x|T(x))P(T(x))} = \frac{P(\theta, T(x))}{P(T(x))} = P(\theta | T(x))$$

Since $$H$$ is a function of $$P$$ we obtain the result.