2
$\begingroup$

I have been trying to calculate the correlation coefficient $(\rho)$ of two variables, and noticed that in cases where either $var(X)$ or $var(Y)$ are very small, the correlation coefficient calculation gives incorrect results, due to the fact that the covariance is coming out too small.

In fact, in the limit where say $x_i = \mu_x$ for all $i$ ($\mu_x$ is the true mean), both the numerator and denominator in the correlation coefficient become zero.

As an example, take $X,Y$ points scattered along a straight line with thickness $\epsilon$. Clearly, as $\epsilon\to0$, $|\rho|\to1$, meaning that one would expect:

$$\left|\text{Cov}(X,Y)\right|\approx \sqrt{\text{var}(X)\text{var}(Y)}$$ De-facto, I'm getting that the covariance is much smaller. As an example, here is an image of pixels for which I want to calculate $\rho$, with the shade of gray giving the weight.

$\quad$enter image description here

My calculations give:

$$\sigma_x = 0.50, \quad \sigma_y = 52.25$$ $$\text{cov(X,Y)} = 4.67 \ll \sqrt{\text{var}(X)\text{var}(Y)} = 26.11$$ Meaning that I get a $\rho$ of $0.18$, while I'd expect $\rho\approx 1$. This problem doesn't occur whenever the line is not oriented to one of the axes, e.g. when the variances are comparable.

Are there any methods for calculating the covariance that overcome this problem?

$\endgroup$
4
  • $\begingroup$ Actually, you need not to compute covariance and variances (since their denominators cancel out). Typically, software programs compute a martrix of correlation coefficients the way shown in footnote here or similar to it. $\endgroup$
    – ttnphns
    Apr 2, 2016 at 14:34
  • $\begingroup$ @ttnphns - care to expound on your comment? $\endgroup$
    – nbubis
    Apr 2, 2016 at 14:40
  • $\begingroup$ @nbubis I have edited my answer, to address the new example you have provided. $\endgroup$ Apr 2, 2016 at 14:44
  • $\begingroup$ How are you actually computing the correlation now? $\endgroup$
    – Glen_b
    Apr 3, 2016 at 9:07

1 Answer 1

4
$\begingroup$

The correlation coefficient, $\rho$ is $$\rho = \dfrac{Cov(X,Y)}{\sqrt{Var(X) Var(Y)}} $$

For numerical stability, you can instead find $\exp(\log(\rho))$.

$$\rho = \exp(\log(\rho)) = \exp\left(\log(Cov(X,Y)) - \dfrac{1}{2}\log(Var(X)) - \dfrac{1}{2} \log(Var(Y) \right). $$

Thus, when the values are really small, $\log$ of the values negative, and thus making them easier to work with.

When the variance is exactly 0 for a random variable, the correlation between the two variables is known to be 0. See here. In your example, the variance along the $Y$ variable is large, but the variance along the $X$ variable is very small. When $\epsilon \to 0$, the correlation does not go to 1!, because when $\epsilon \to 0$, $Var(X) \to 0$.

Intuitively, you can understand it in this way. Correlation tells you how much the change in $X$ will impact the linear change in $Y$, When there is no variability in $X$, then there is no change in $X$, so it cannot impact any linear change in $Y$!

When your pixel cloud is oriented in the non standard axis, then there is positive variation in both $X$ and $Y$, and thus you expect the correlation to be 1 then.

$\endgroup$
4
  • $\begingroup$ This is a nice idea, but unfortunately doesn't solve the underlying issue, which is in the covariance calculation. I tried implementing this and the results are quite close to what I was getting previously. $\endgroup$
    – nbubis
    Apr 2, 2016 at 14:00
  • $\begingroup$ Please include the issue with the covariance in the original question. If this doesn't solve your problem, then we will need more information about your problem. $\endgroup$ Apr 2, 2016 at 14:04
  • $\begingroup$ I added a numerical example, which I think illustrates the issue. Thanks! $\endgroup$
    – nbubis
    Apr 2, 2016 at 14:34
  • $\begingroup$ Thank you! I guess my problem wasn't calculation, but rather a basic misunderstanding. Oh well :) $\endgroup$
    – nbubis
    Apr 2, 2016 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.