I was trying to prove that sufficient statistics attain equality in the data processing inequality by a slightly different route than I usually see, and came across an odd expression. (I care more about understanding the odd expression than about proving the data processing inequality).

So if we have $\theta \longrightarrow X \longrightarrow T(X)$, the data processing inequality gives us

$$I(\theta;X) \ge I(\theta;T(X))\tag{1}$$

and expanding out into entropies yields $H(\theta) - H(\theta\mid X) \ge H(\theta) - H(\theta\mid T(X))$, which yields $$H(\theta\mid X) \le H(\theta\mid T(X))\tag{2}$$

So $(1)$ attains equality iff $(2)$ does. But how on earth do I even interpret $(2)$? $T(X)$ is generally a non-injective mapping, and so it seems very strange to me that a non-injective mapping (which should strictly decrease discrete entropy) could possibly lead to an equality.

I was trying to manipulate the Neyman-Fisher factorization to try and flip into the form of $(2)$, but to no avail.

Any thoughts?



Yes, it is true that if $T$ is sufficient, then $$H(\theta | x) = H(\theta|T(x))$$.

From the definition of sufficiency, we have that $$P(x|T(x), \theta) = P(x|T(x))$$.

And now

$$P(\theta | x) = \frac{P(x, \theta)}{P(x)} = \frac{P(x, T(x), \theta)}{P(x, T(x))} = \frac{P(x|T(x), \theta)P(\theta, T(x))}{P(x|T(x))P(T(x))}$$

Now we use the above fact to simplify:

$$\frac{P(x|T(x), \theta)P(\theta, T(x))}{P(x|T(x))P(T(x))} = \frac{P(\theta, T(x))}{P(T(x))} = P(\theta | T(x))$$

Since $H$ is a function of $P$ we obtain the result.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.