I'm interested in the mutual information of two continuous random variables $X$ and $Y$. Shannon defined differential entropy as $h(X) = -\int p_X(x)\log p_X(x) dx$, where $p_X$ is the probability density function of $X$. He obtained this formula by swapping the sum for an integral and the mass function for the density function in his well-derived formula for discrete random variable entropy.

However, differential entropy turned out not to be the correct continuous analogue. Jaynes showed that it should actually be

$\lim_{N\rightarrow \infty}H_N(X)=\log(N)-\int p_X(x)\log\frac{p_X(x)}{m(x)}dx$

where $m(x)$ is an invariant measure. See https://en.wikipedia.org/wiki/Limiting_density_of_discrete_points. This seems a bit strange since $N$ on LHS is within a limit, but not on RHS (if someone can explain, that would be great!).

It's also sometimes written just $H(X)=-\int p_X(x)\log\frac{p_X(x)}{m(x)}dx$

This fixes differential entropy's problems of being sometimes negative and not invariant under change of variables.

I'm wondering whether continuous mutual information has a similar problem to differential entropy. It's defined:

$I(X;Y) = \int\int p_{X,Y}(x,y)\log\frac{p_{X,Y}(x,y)}{p_X(x)p_Y(y)}dxdy = h(X)+h(Y)-h(X,Y)$

where $p_{X,Y}$ is the joint density and $h(X,Y)$ is the joint continuous entropy. As far as I can tell, there are no major problems with continuous mutual information. I'm still curious about whether it is the correct continuous analogue of discrete mutual information though. I think that it is, but I haven't learnt enough measure theory to be sure.

According to the link above, $H_N(X) \approx log(N)-log(r)+h(X)$, when $m(x)$ is constant over an interval of size $r$ and $p_X(x)$ is essentially zero outside that interval, which makes some intuitive sense I think. With a little manipulation, this gives $H_N(X) + H_M(Y) - H_{NM}(X,Y) \approx h(X) + h(Y) - h(X,Y) = I(X;Y)$, I think, which would be nice.

Alternatively, through some algebraic manipulation directly from definitions, we can see that $ \begin{align} H(X)+H(Y)-H(X,Y) & = h(X)+h(Y)-h(X,Y)+\int p_X(x)\log m_X(x) dx \\ & \ \ \ + \int p_Y(y)\log m_Y(y) dy - \int\int p_{X,Y}(x,y)\log m_{X,Y}(x,y) \\ &= I(X;Y)+\int p_X(x)\log m_X(x) dx+ \int p_Y(y)\log m_Y(y) dy \\ & \ \ \ - \int\int p_{X,Y}(x,y)\log m_{X,Y}(x,y)dxdy \end{align} $ where $m_X$, $m_Y$ and $m_{X,Y}$ are invariant measures. I'm wondering if the last bit on RHS cancels nicely so that both versions of entropy give the same version of mutual information.

Is continuous mutual information the correct analogue of discrete mutual information and am I on the right track to proving it, if so? If the two versions of entropy yield different notions of mutual information, what's the difference between them? Is one better than the other in some contexts?

Thanks :)

  • $\begingroup$ I think the extra stuff on RHS does cancel actually (with some assumptions). Assuming the invariant measure is constant on an interval of size $r$, we get $\int p_X(x)\log m_X(x)dx = - \log(r)$ right? $\endgroup$ – DM-97 Aug 3 '18 at 17:00
  • $\begingroup$ Then everything should cancel nicely: $\log(r_Xr_Y) - \log r_X - \log r_Y = 0$ $\endgroup$ – DM-97 Aug 3 '18 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.