I am confused how to interpret the result of performing a normalized correlation with a constant vector. Since you have to divide by the standard deviation of both vectors (reference: http://en.wikipedia.org/wiki/Cross-correlation ), if one of them is constant (say a vector of all 5's, which has standard deviation of zero), then the correlation is infinity, but in fact the correlation should be zero right? This isn't just a corner case, in general if the standard deviation of one of the vectors is small, the correlation to any other vector is very high, which obviously doesn't make sense. Can anyone explain my misinterpretation?
$\begingroup$
$\endgroup$
6
-
2$\begingroup$ Matters are much worse than you think. The correlation is proportional to the covariance which is estimated from $$\sum_i (x_i - \bar{x})(y_i - \bar{y}) = \sum_i (5 - \bar{5})(y_i - \bar{y}) = 0,$$ (since the average value of $5$ is $\bar{5}=5$) and so when you divide by the standard deviation of $5$, you get the indeterminate form $\frac{0}{0}$ for the correlation. $\endgroup$– Dilip SarwateCommented Feb 26, 2012 at 3:19
-
$\begingroup$ I noticed a few typos and the link to Wikipedia was broken. I tried to guess the right one, but if I got it wrong, please edit to correct. Welcome to the site. $\endgroup$– cardinalCommented Feb 26, 2012 at 3:27
-
2$\begingroup$ Perhaps thinking about this geometrically will help. Assuming the sample standard deviations are both positive (even if "small"), let $u_i = \frac{1}{\sqrt{n}} (x_i-\bar x) / s_x$ and $v_i = \frac{1}{\sqrt{n}} (y_i - \bar y) / s_y$ where $s_x$ and $s_y$ are the respective standard deviations. Then $\mathbf u = (u_i)$ and $\mathbf v = (v_i)$ are just unit vectors and the correlation $\rho = \mathbf u^T \mathbf v$ is just the cosine of the angle between them. This tells you (almost) everything you need to know about the values that can be taken on. $\endgroup$– cardinalCommented Feb 26, 2012 at 3:32
-
1$\begingroup$ (For statistical reasons, slightly different normalizing factors often show up, but hopefully the intuition is clear. If you're still worried about "small" standard deviations, ask yourself how the standard deviation and the correlation each change, respectively, if you replace $x_i$ by $a x_i + b$, for example.) $\endgroup$– cardinalCommented Feb 26, 2012 at 3:34
-
1$\begingroup$ I still don't follow. Do we consider the indeterminate case to mean "not correlated"? Let's take 3 vectors: a = [1 9 1 9 1], b = [9 1 9 1 9], c = [5 5 5 5 5]. a and b should have some negative correlation. I would have said that (a and c) and also (a and b) had zero correlation, but I guess we have seen that they actually have "indeterminate" correlation. How would you check for this in software? Just set the correlation to zero if either the numerator or denominator are < epsilon (say 1e-5 or something)? Isn't there some quantity that would behave more like I am expecting? $\endgroup$– David DoriaCommented Feb 26, 2012 at 13:17
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
Let $\boldsymbol{x}$ and $\boldsymbol{y}$ be your two vectors and let $\boldsymbol{\bar{x}} \equiv \bar{x} \boldsymbol{1}$ and $\boldsymbol{\bar{y}} \equiv \bar{y} \boldsymbol{1}$ be constant vectors for the means of the two original vectors. The components of the sample correlation are:
$$\begin{matrix} s_{x,y}^2 = \frac{1}{n-1} (\boldsymbol{x} - \boldsymbol{\bar{x}}) \cdot (\boldsymbol{y} - \boldsymbol{\bar{y}}) & & s_x = \frac{1}{n-1} ||\boldsymbol{x} - \boldsymbol{\bar{x}}|| & & s_y = \frac{1}{n-1} ||\boldsymbol{y} - \boldsymbol{\bar{y}}||. \end{matrix}$$
The sample correlation between $\boldsymbol{x}$ and $\boldsymbol{y}$ is just the cosine of the angle between the vectors $\boldsymbol{x} - \boldsymbol{\bar{x}}$ and $\boldsymbol{y} - \boldsymbol{\bar{y}}$. Letting this angle be $\theta$ we have:
$$\rho_{x,y} = \frac{(\boldsymbol{x} - \boldsymbol{\bar{x}}) \cdot (\boldsymbol{y} - \boldsymbol{\bar{y}})}{||\boldsymbol{x} - \boldsymbol{\bar{x}}|| \cdot ||\boldsymbol{y} - \boldsymbol{\bar{y}}||} = \cos \theta.$$
Since scaling of either vector scales the covariance and standard deviation equivalently, this means that correlation is unaffected by scale. It is not correct to say that a low standard deviation gives a high correlation. What matters for correlation is the angle between the vectors, not their lengths.
In the special case where $\boldsymbol{y} \propto \boldsymbol{1}$ (i.e., $\boldsymbol{y}$ is a constant vector) you have $\boldsymbol{y} - \boldsymbol{\bar{y}} = \boldsymbol{0}$ which then gives $s_{x,y}^2 = 0$ and $s_{y} = 0$. In this case the correlation is undefined. Geometrically this occurs because there is no defined angle with the zero vector.
