Let's say $X$ and $Y$ are two random variables where $Y \sim N(0, \sigma) $ and independent from $X$. Is there a general way to express the entropy $H(X+Y)$ in terms $H(X)$? $X$ can be any random variable.
$\begingroup$
$\endgroup$
9
-
$\begingroup$ What does "$P$" mean? What do you know or assume about $X$? $\endgroup$– whuber ♦Commented Dec 7, 2017 at 22:42
-
$\begingroup$ Sorry that my question was unclear, edited $\endgroup$– WezCommented Dec 7, 2017 at 22:45
-
$\begingroup$ Thank you. Are you sure this is what you want ask? Or are you trying to ask about the entropy of the random variable $X+Y$ itself? Please note that $P(X+Y)$ is a random variable, but its entropy will likely differ quite a bit from that of $X+Y.$ $\endgroup$– whuber ♦Commented Dec 7, 2017 at 22:47
-
1$\begingroup$ OK. Are you familiar with how entropy behaves with sums of independent variables? See en.wikipedia.org/wiki/…. $\endgroup$– whuber ♦Commented Dec 7, 2017 at 22:53
-
$\begingroup$ As far as I understand, I can express $H(X,Y)=H(X)+H(Y)$ if $X$ and $Y$ are independent $\endgroup$– WezCommented Dec 7, 2017 at 22:56
|
Show 4 more comments
2 Answers
$\begingroup$
$\endgroup$
This is like sending a signal, $X$, through and additive white Gaussian noise channel with variance, $\sigma^2$, where $Z=X+Y$ is the received signal with noise. One thing we can say is that if the $E(X^2) = P$, then $I(X;Z) \leq 1/2 \log (1+P/\sigma^2)$. That is relevant because $I(X;Z) = H(Z)-H(Z|X) = H(X+Y)-H(Z|X) = H(X+Y) - 1/2 \log (2 \pi e \sigma^2)$, and therefore $H(X+Y) \leq 1/2 \log (1+P/\sigma^2) + 1/2 \log (2 \pi e \sigma^2) = 1/2 \log (2 \pi e (\sigma^2+P))$.
That is not very satisfying as it only gives an upper bound and it is in terms of the second moment, not the entropy of $X$. One thing we can see from this bound is that it is saturated if $X$ is normal. If, on the other hand, $X$ is a delta function (e.g. X = 0, always), then $H(X+Y) = 1/2 \log (2 \pi e \sigma^2)$ even though $H(X)=-\infty$. I don't think there will be a general relationship, as I found many papers deriving relationships when $X,Y$ were in specific parametric families.
$\begingroup$
$\endgroup$
0
This has been bothering me since I wrote my last answer. There's a much better answer based on the entropy power inequality, but I'll leave the old answer in case it's useful to anybody. The entropy power, $e^{2/n H(X)}$, where $n$ is the dimension of the random variable and H is differential entropy, is also the volume of the typical set for the distribution p(x). It seems intuitive that adding two independent random variables should increase the size of the typical set. This is reflected in the entropy power inequality, which can be rearranged as follows. $$H(X+Y) \geq n/2 \log (e^{2/n H(X)}+e^{2/n H(Y)})$$
In your case H(Y) is known and Gaussian. As the noise variance goes to zero the lower bound goes to H(X), as expected.
