I am confused how to interpret the result of performing a normalized correlation with a constant vector. Since you have to divide by the standard deviation of both vectors (reference: http://en.wikipedia.org/wiki/Cross-correlation ), if one of them is constant (say a vector of all 5's, which has standard deviation of zero), then the correlation is infinity, but in fact the correlation should be zero right? This isn't just a corner case, in general if the standard deviation of one of the vectors is small, the correlation to any other vector is very high, which obviously doesn't make sense. Can anyone explain my misinterpretation?

  • 2
    $\begingroup$ Matters are much worse than you think. The correlation is proportional to the covariance which is estimated from $$\sum_i (x_i - \bar{x})(y_i - \bar{y}) = \sum_i (5 - \bar{5})(y_i - \bar{y}) = 0,$$ (since the average value of $5$ is $\bar{5}=5$) and so when you divide by the standard deviation of $5$, you get the indeterminate form $\frac{0}{0}$ for the correlation. $\endgroup$ – Dilip Sarwate Feb 26 '12 at 3:19
  • $\begingroup$ I noticed a few typos and the link to Wikipedia was broken. I tried to guess the right one, but if I got it wrong, please edit to correct. Welcome to the site. $\endgroup$ – cardinal Feb 26 '12 at 3:27
  • 1
    $\begingroup$ Perhaps thinking about this geometrically will help. Assuming the sample standard deviations are both positive (even if "small"), let $u_i = \frac{1}{\sqrt{n}} (x_i-\bar x) / s_x$ and $v_i = \frac{1}{\sqrt{n}} (y_i - \bar y) / s_y$ where $s_x$ and $s_y$ are the respective standard deviations. Then $\mathbf u = (u_i)$ and $\mathbf v = (v_i)$ are just unit vectors and the correlation $\rho = \mathbf u^T \mathbf v$ is just the cosine of the angle between them. This tells you (almost) everything you need to know about the values that can be taken on. $\endgroup$ – cardinal Feb 26 '12 at 3:32
  • 1
    $\begingroup$ (For statistical reasons, slightly different normalizing factors often show up, but hopefully the intuition is clear. If you're still worried about "small" standard deviations, ask yourself how the standard deviation and the correlation each change, respectively, if you replace $x_i$ by $a x_i + b$, for example.) $\endgroup$ – cardinal Feb 26 '12 at 3:34
  • 1
    $\begingroup$ I still don't follow. Do we consider the indeterminate case to mean "not correlated"? Let's take 3 vectors: a = [1 9 1 9 1], b = [9 1 9 1 9], c = [5 5 5 5 5]. a and b should have some negative correlation. I would have said that (a and c) and also (a and b) had zero correlation, but I guess we have seen that they actually have "indeterminate" correlation. How would you check for this in software? Just set the correlation to zero if either the numerator or denominator are < epsilon (say 1e-5 or something)? Isn't there some quantity that would behave more like I am expecting? $\endgroup$ – David Doria Feb 26 '12 at 13:17

Let $\boldsymbol{x}$ and $\boldsymbol{y}$ be your two vectors and let $\boldsymbol{\bar{x}} \equiv \bar{x} \boldsymbol{1}$ and $\boldsymbol{\bar{y}} \equiv \bar{y} \boldsymbol{1}$ be constant vectors for the means of the two original vectors. The components of the sample correlation are:

$$\begin{matrix} s_{x,y}^2 = (\boldsymbol{x} - \boldsymbol{\bar{x}}) \cdot (\boldsymbol{y} - \boldsymbol{\bar{y}}) & & s_x = ||\boldsymbol{x} - \boldsymbol{\bar{x}}|| & & s_y = ||\boldsymbol{y} - \boldsymbol{\bar{y}}||. \end{matrix}$$

The sample correlation between $\boldsymbol{x}$ and $\boldsymbol{y}$ is just the cosine of the angle between the vectors $\boldsymbol{x} - \boldsymbol{\bar{x}}$ and $\boldsymbol{y} - \boldsymbol{\bar{y}}$. Letting this angle be $\theta$ we have:

$$\rho_{x,y} = \frac{(\boldsymbol{x} - \boldsymbol{\bar{x}}) \cdot (\boldsymbol{y} - \boldsymbol{\bar{y}})}{||\boldsymbol{x} - \boldsymbol{\bar{x}}|| \cdot ||\boldsymbol{y} - \boldsymbol{\bar{y}}||} = \cos \theta.$$

Since scaling of either vector scales the covariance and standard deviation equivalently, this means that correlation is unaffected by scale. It is not correct to say that a low standard deviation gives a high correlation. What matters for correlation is the angle between the vectors, not their lengths.

In the special case where $\boldsymbol{y} \propto \boldsymbol{1}$ (i.e., $\boldsymbol{y}$ is a constant vector) you have $\boldsymbol{y} - \boldsymbol{\bar{y}} = \boldsymbol{0}$ which then gives $s_{x,y}^2 = 0$ and $s_{y} = 0$. In this case the correlation is undefined. Geometrically this occurs because there is no defined angle with the zero vector.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.