Consider a random variable $X$ to be uniformly distributed in a domain $\Omega$, $$P(x) = 1/|\Omega|$$, with $|\Omega| = \int_\Omega dx$.

Consider a second random variable defined by $Y=f(X)$, such that $P(y|x) = \delta(y - f(x))$.

The marginal of $y$ is $$P(y) = \int_\Omega P(y|x)P(x) dx = \frac{1}{|\Omega|}\int_\Omega \delta(y - f(x)) dx$$

The entropy of $X$ is given by $$S(X) = -\int_\Omega P(x) \log P(x) dx = \log |\Omega|$$

My question is how to compute the entropy of X given Y (note that this is not the conditional entropy; it depends on y),

$$S(X|y) = -\int_\Omega P(x|y) \log P(x|y) dx$$

I'm expecting it to be $S(X|y) = |\Omega|P(y) = \int_\Omega \delta(y - f(x))dx$, but I'm not being able to get it.

What I've tried so far:

Using $$P(X|Y) = \frac{P(Y|X)P(X)}{P(Y)} = \frac{\delta(y - f(x))}{|\Omega|P(y)}$$, $S(X|y)$ reduces to

$$ S(X|y) = \log |\Omega|P(y) -\frac{1}{|\Omega| P(y)}\int_\Omega \delta(y - f(x)) \log \delta(y - f(x)) dx $$

The first term is what I would expect. But I'm not being able to show that the second term is 0. What am I missing?

[This question was originally posted in the context of physics, but it seems the difficulty is on the stats.]


It depends on $f$. If $f$ is injective (so $f^{-1}$ is well-defined) then $X$ can take only one value, given $y$, namely $f^{-1}(y)$ -- so the entropy of $X$ is zero. At the other extreme, if $f$ maps everything to a constant then $y$ conveys no information about $X$ and so $S(X|y)=S(X)$.

  • 1
    $\begingroup$ Your reasoning is qualitatively correct, and the question here is how to compute $S(X|y)$ and quantify that information when $f$ is neither injective nor the constant function. $\endgroup$ – Jorge Leitao Dec 16 '15 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.