Z-transforming Pearson correlation vs. converting to mutual information: they seem to be related, but how?

Question

EDIT: By noodling around a bit, I found that they can actually both be re-arranged into even more similar forms:

$Z = -\frac{1}{2}(\ln(1-\rho) - \ln(1+\rho))$

$I =-\frac{1}{2}(\ln(1-\rho) + \ln(1+\rho))$

This seems too similar to be just a coincidence.

Original question:

Given a Pearson correlation coefficient ρ, it is common to transform it using Fisher's Z-transform:

$z = \frac{1}{2}\ln(\frac{1+\rho}{1-ρ})$

I noticed that this is extremely similar to the formula for the conversion of a Pearson correlation coefficient to the mutual information (assuming only linear relationships):

$I = -\frac{1}{2}\ln(1-ρ^2)$

They are clearly of a similar form, but it's not entirely clear how the relate to each-other. It seems like the FIsher's Z is doing something like turning into the correlation into a "quasi-MI."

Does anyone have an intuitive explanation for what is happening here? Or know of any literature I can consult to dig into more of this?

Answer 1

So the equation that you presented for mutual information is valid only in the case that both $X$ and $Y$ are normally distributed. Theorem 2.1 of Gelfand and Yaglom (1957) proves this relationship. As noted in this paper the relationship between the correlation coefficient and the mutual information makes sense because both quantities seek to describe the joint dependence structure of $X$ and $Y$. Indeed, the two metrics are complimentary, both revealing different properties of the linear mutual dependence structure of the two random variables. This is clear from the functional form,

$$I(X,Y)=-0.5 \ln(1-\rho^2)$$

This shows that the mutual information goes to infinity if $\rho=\pm 1$. Thus, when there is complete covariance of the two variables. Note that Pearson's un-standardized correlation coefficient is essentially trying to capture the covariance. This relationship and its relationship to mutual information are further discussed in this previous answer. Essentially, the correlation coefficient is trying to quantify the linear correlation-covariance of $X$ and $Y$. The mutual information is essentially measuring how "non-independent" $X$ and $Y$ are. Under the Gaussian assumption, this measure boils down to a function of the linear relationship (Gelfand and Yaglom, 1957).

Furthermore, since you mention the Fisher transformation I figure I should shed some light on why we do this procedure. This transformation is essentially a variance stabilizing transformation. The sample correlation coefficient is given by,

$$r = \frac{\sum_i (X_i-\bar X)(Y_i - \bar Y)}{[\sum_i (X_i-\bar X)^2(Y_i - \bar Y)^2]^{1/2}}$$

Using the delta method it is possible (assuming some existence conditions for higher-order moments) to show that,

$$\sqrt{n}(r-\rho)\overset{d}{\to}N(0,(1-\rho^2)^2)$$

Notice that that variance depends on $\rho$ here. This is a rather undesirable trait especially if we want to build confidence sets since we do not know $\rho$. So what we do instead is to find a transformation, $\phi$, of the parameter that stabilizes the asymptotic variance so that we may build confidence sets. We want to find $\phi$ such that the asymptotic variance is constant and so we can usually calculate it as,

$$\phi(\theta)=\int \frac{1}{\sigma(\theta)}d\theta$$

Thus, the variance stabilizing transformation can be computed as,

$$\phi(r)=\int\frac{1}{1-r^2}d\rho=\frac{1}{2}\ln\frac{1+r}{1-r}$$

This is actually the arctanh function and it is the Fisher transformation. Notice that just like the Gaussian mutual information this function will approach positive or negative infinity when $r=\pm 1$.