Connection between $R^2$ and Mutual Information

Question

Interpreting MI as "the amount of uncertainty removed by adding some information", I would expect some connection to $R^2$, which is the "amount of variance explained by adding some information".

In particular, $I(X,Y) = H(Y)-H(Y|X)$, which, poorly rephrased, is, "how uncertain you are about $Y$ to begin with" - "how uncertain you are about $Y$ after you got to know $X$"

On the other hand, $R^2 = 1-\frac{\sum (y_i - f(x_i))^2}{\sum (y_i - \bar{y})^2}$, which, to me, maybe wrongly, is sort of assuming that $p(Y|X) \sim N(0, \sigma_1^2)$ where $\sigma_1^2 = N^{-1}\sum (y_i - f(x_i))^2$ and $p(Y)\sim N(0, \sigma_2^2)$ where $\sigma_2^2 = N^{-1}\sum (y_i - \bar{y})^2$... which leaves us with $R^2 = 1-E_x[Var[p(y|x)]]/Var[p(y)]$

Now, if we define $\hat{R^2} = \frac{Var[p(y)]}{E_x[Var[p(y|x)]]} \propto R^2$ (proportional in a non-linear way), I'd say it looks like that:

$$ I(X,Y) \stackrel{?}{=} \alpha\ln \hat{R^2} = \alpha(\ln(Var[p(y)]) -\ln(E_x[Var[p(y|x)]])),\,\, \text{for some }\alpha \in \mathbb{R}^+ $$

And indeed, the entropy of a Gaussian distribution is $\ln c\sigma^2$

So in other words, it seems like there is indeed a connection between $R^2$ (or at least, its made up brother $\hat{R^2}$) and Mutual Information, if we assume $p(y)$ and $p(y|x)$ to be Gaussian

Am I doing something wrong/saying something that is just made up?

Answer 1

Firstly, mutual information is a measure of statistical dependence between random variables, whereas $R^2$ is a measure of goodness-of-fit. In other words, $I(X;Y)$ is intrinsic to the data, whereas $R^2$ is defined with respect to a predictive model ($f$ in your notation). So they aren't quite directly comparable.

However, for the case of simple linear regression, if $X, Y$ are jointly Gaussian, $R^2$ and $I(X; Y)$ are related.

For Gaussian random variables with correlation $\rho$

$$I(X; Y) = -\frac{1}{2} \log (1-\rho^2)$$

(see Thomas & Cover chapter 8, eqn. 8.52 for details)

And the $R^2$ should be $\rho^2$.

More generally, $I(X;Y)$ can be used to construct an upper bound on $R^2$. This reasoning applies for non-Gaussian data and any choice of $f$. Starting from your observation that

$$R^2 = 1 - \frac{\mathbb{E}[(Y-f(X))^2]}{\mathbb{E}[(Y - \mathbb{E}(Y))^2]}$$

$$\leq 1 - \frac{\text{Var}(Y|X)}{\text{Var}(Y)}$$

we can note that the maximum entropy distribution with fixed variance is a Gaussian, so

$$R^2 \leq 1 - \frac{e^{2 h(Y|X)}}{2\pi e \text{Var}(X)}$$

$$= 1 - \frac{e^{2(h(Y) - I(X;Y))}}{2\pi e \text{Var}(X)}$$

Hope this helps!!