Skip to main content
Cross Validated

Return to Answer

edited body
Source Link
Roger V.
Roger V.
  • 4.5k
  • 15
  • 35

Let's suppose that $Y=f(X)+\epsilon Z$ where $Z$ is a standard normal, and we take the limit $\epsilon\rightarrow 0$

The mutual information equals $$I = H(Y) - H(Y|X)$$,$$I = H(Y) - H(Y|X),$$ where $H(Y)$ is the entropy of $Y$ and $H(Y|X)$ is the conditional entropy of $Y$ on $X$. Since $Z$ is normal, $H(Y|X)=\frac{1}{2}\ln(2\pi e\epsilon^2)$.

If $f$ is not constant and smooth, then $H(Y)$ stays positive and finite. But as $\epsilon\rightarrow 0$ we find $H(Y|X)\rightarrow-\infty$ and therefore $I=\infty$ in this limit.

If $f(X)=c$ is a constant, then $H(Y)=H(Y|X)$ for any positive $\epsilon$, and therefore in this case the limit is $I=0$.

Let's suppose that $Y=f(X)+\epsilon Z$ where $Z$ is a standard normal, and we take the limit $\epsilon\rightarrow 0$

The mutual information equals $$I = H(Y) - H(Y|X)$$, where $H(Y)$ is the entropy of $Y$ and $H(Y|X)$ is the conditional entropy of $Y$ on $X$. Since $Z$ is normal, $H(Y|X)=\frac{1}{2}\ln(2\pi e\epsilon^2)$.

If $f$ is not constant and smooth, then $H(Y)$ stays positive and finite. But as $\epsilon\rightarrow 0$ we find $H(Y|X)\rightarrow-\infty$ and therefore $I=\infty$ in this limit.

If $f(X)=c$ is a constant, then $H(Y)=H(Y|X)$ for any positive $\epsilon$, and therefore in this case the limit is $I=0$.

Let's suppose that $Y=f(X)+\epsilon Z$ where $Z$ is a standard normal, and we take the limit $\epsilon\rightarrow 0$

The mutual information equals $$I = H(Y) - H(Y|X),$$ where $H(Y)$ is the entropy of $Y$ and $H(Y|X)$ is the conditional entropy of $Y$ on $X$. Since $Z$ is normal, $H(Y|X)=\frac{1}{2}\ln(2\pi e\epsilon^2)$.

If $f$ is not constant and smooth, then $H(Y)$ stays positive and finite. But as $\epsilon\rightarrow 0$ we find $H(Y|X)\rightarrow-\infty$ and therefore $I=\infty$ in this limit.

If $f(X)=c$ is a constant, then $H(Y)=H(Y|X)$ for any positive $\epsilon$, and therefore in this case the limit is $I=0$.

deleted 26 characters in body
Source Link
a06e
a06e
  • 4.6k
  • 2
  • 24
  • 52

Let's suppose that $Y=f(X)+\epsilon Z$ where $Z$ is a standard normal, and we take the limit $\epsilon\rightarrow 0$

The mutual information equals $$I = H(Y) - H(Y|X)$$, where $H(Y)$ is the entropy of $Y$ and $H(Y|X)$ is the conditional entropy of $Y$ on $X$. Since $Z$ is normal, $H(Y|X)=\frac{1}{2}\ln(2\pi e\epsilon^2)$.

If $f$ is not constant and smooth, then $H(Y)$ isstays positive and finite. AsBut as $\epsilon\rightarrow 0$ we find $H(Y|X)\rightarrow-\infty$ and therefore $I=\infty$ in this limit.

If $f(X)=c$ is a constant, then $I=H(Y)=-\infty$$H(Y)=H(Y|X)$ for any positive $\epsilon$, and therefore in this case the limit is $I=-\infty$ (and not zero as proposed in the OP)$I=0$.

Let's suppose that $Y=f(X)+\epsilon Z$ where $Z$ is a standard normal, and we take the limit $\epsilon\rightarrow 0$

The mutual information equals $$I = H(Y) - H(Y|X)$$, where $H(Y)$ is the entropy of $Y$ and $H(Y|X)$ is the conditional entropy of $Y$ on $X$. Since $Z$ is normal, $H(Y|X)=\frac{1}{2}\ln(2\pi e\epsilon^2)$.

If $f$ is not constant and smooth, then $H(Y)$ is finite. As $\epsilon\rightarrow 0$ we find $H(Y|X)\rightarrow-\infty$ and therefore $I=\infty$ in this limit.

If $f(X)=c$ is a constant, then $I=H(Y)=-\infty$ for any positive $\epsilon$, and therefore in this case the limit is $I=-\infty$ (and not zero as proposed in the OP).

Let's suppose that $Y=f(X)+\epsilon Z$ where $Z$ is a standard normal, and we take the limit $\epsilon\rightarrow 0$

The mutual information equals $$I = H(Y) - H(Y|X)$$, where $H(Y)$ is the entropy of $Y$ and $H(Y|X)$ is the conditional entropy of $Y$ on $X$. Since $Z$ is normal, $H(Y|X)=\frac{1}{2}\ln(2\pi e\epsilon^2)$.

If $f$ is not constant and smooth, then $H(Y)$ stays positive and finite. But as $\epsilon\rightarrow 0$ we find $H(Y|X)\rightarrow-\infty$ and therefore $I=\infty$ in this limit.

If $f(X)=c$ is a constant, then $H(Y)=H(Y|X)$ for any positive $\epsilon$, and therefore in this case the limit is $I=0$.

Source Link
a06e
a06e
  • 4.6k
  • 2
  • 24
  • 52

Let's suppose that $Y=f(X)+\epsilon Z$ where $Z$ is a standard normal, and we take the limit $\epsilon\rightarrow 0$

The mutual information equals $$I = H(Y) - H(Y|X)$$, where $H(Y)$ is the entropy of $Y$ and $H(Y|X)$ is the conditional entropy of $Y$ on $X$. Since $Z$ is normal, $H(Y|X)=\frac{1}{2}\ln(2\pi e\epsilon^2)$.

If $f$ is not constant and smooth, then $H(Y)$ is finite. As $\epsilon\rightarrow 0$ we find $H(Y|X)\rightarrow-\infty$ and therefore $I=\infty$ in this limit.

If $f(X)=c$ is a constant, then $I=H(Y)=-\infty$ for any positive $\epsilon$, and therefore in this case the limit is $I=-\infty$ (and not zero as proposed in the OP).