# How to compute the entropy of the specific conditional probability

Consider a random variable $X$ to be uniformly distributed in a domain $\Omega$, $$P(x) = 1/|\Omega|$$, with $|\Omega| = \int_\Omega dx$.

Consider a second random variable defined by $Y=f(X)$, such that $P(y|x) = \delta(y - f(x))$.

The marginal of $y$ is $$P(y) = \int_\Omega P(y|x)P(x) dx = \frac{1}{|\Omega|}\int_\Omega \delta(y - f(x)) dx$$

The entropy of $X$ is given by $$S(X) = -\int_\Omega P(x) \log P(x) dx = \log |\Omega|$$

My question is how to compute the entropy of X given Y (note that this is not the conditional entropy; it depends on y),

$$S(X|y) = -\int_\Omega P(x|y) \log P(x|y) dx$$

I'm expecting it to be $S(X|y) = |\Omega|P(y) = \int_\Omega \delta(y - f(x))dx$, but I'm not being able to get it.

What I've tried so far:

Using $$P(X|Y) = \frac{P(Y|X)P(X)}{P(Y)} = \frac{\delta(y - f(x))}{|\Omega|P(y)}$$, $S(X|y)$ reduces to

$$S(X|y) = \log |\Omega|P(y) -\frac{1}{|\Omega| P(y)}\int_\Omega \delta(y - f(x)) \log \delta(y - f(x)) dx$$

The first term is what I would expect. But I'm not being able to show that the second term is 0. What am I missing?

[This question was originally posted in the context of physics, but it seems the difficulty is on the stats.]

It depends on $f$. If $f$ is injective (so $f^{-1}$ is well-defined) then $X$ can take only one value, given $y$, namely $f^{-1}(y)$ -- so the entropy of $X$ is zero. At the other extreme, if $f$ maps everything to a constant then $y$ conveys no information about $X$ and so $S(X|y)=S(X)$.
• Your reasoning is qualitatively correct, and the question here is how to compute $S(X|y)$ and quantify that information when $f$ is neither injective nor the constant function. – Jorge Leitao Dec 16 '15 at 14:53