How would you explain the difference between correlation and covariance?

Question

Following up on this question, How would you explain covariance to someone who understands only the mean?, which addresses the issue of explaining covariance to a lay person, brought up a similar question in my mind.

How would one explain to a statistics neophyte the difference between covariance and correlation? It seems that both refer to the change in one variable linked back to another variable.

Similar to the referred-to question, a lack of formulae would be preferable.

Answer 1

The problem with covariances is that they are hard to compare: when you calculate the covariance of a set of heights and weights, as expressed in (respectively) meters and kilograms, you will get a different covariance from when you do it in other units (which already gives a problem for people doing the same thing with or without the metric system!), but also, it will be hard to tell if (e.g.) height and weight 'covary more' than, say the length of your toes and fingers, simply because the 'scale' the covariance is calculated on is different.

The solution to this is to 'normalize' the covariance: you divide the covariance by something that represents the diversity and scale in both the covariates, and end up with a value that is assured to be between -1 and 1: the correlation. Whatever unit your original variables were in, you will always get the same result, and this will also ensure that you can, to a certain degree, compare whether two variables 'correlate' more than two others, simply by comparing their correlation.

Note: the above assumes that the reader already understands the concept of covariance.

Answer 2

The requirements of these types of questions strike me as a bit bizarre. Here is a mathematical concept/formula, yet I want to talk about it in some context completely devoid of mathematical symbols. I also think it should be stated that the actual algebra necessary to understand the formulas, I would think, should be taught to most individuals before higher education (no understanding of matrix algebra is needed, just simple algebra will suffice).

So, at first instead of completely ignoring the formula and speaking of it in some magical and heuristic types of analogies, lets just look at the formula and try to explain the individual components in small steps. The difference in terms of covariance and correlation, when looking at the formulas, should become clear. Whereas speaking in terms of analogies and heuristics I suspect would obsfucate two relatively simple concepts and their differences in many situations.

So lets starts out with a formula for the sample covariance (these I have just taken and adopted from wikipedia);

$\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})$

To get everyone up to speed, lets explicitly define all of the elements and operations in the formula.

• $x_i$ and $y_i$ are each measurements of two seperate attributes of the same observation
• $\bar{x}$ and $\bar{y}$ are the means (or average) of each attribute
• For $\frac{1}{n-1}$, lets just say this means we divide the final result by ${n-1}$.
• $\sum_{i=1}^{n}$ may be a foreign symbol to some, so it would likely be useful to explain this operation. It is simply the sum of all $i$ seperate observations, and $n$ represents the total number of observations.

At this point, I might introduce a simple example, to put a face on the elements and operations so to speak. So for example, lets just make up a table, where each row corresponds to an observation (and $x$ and $y$ are labeled appropriately). One would likely make these examples more specific (e.g. say $x$ represents age and $y$ represents weight), but for our discussion here it should not matter.

x y
---
2 5
4 8
9 3
5 6
0 8

At this point if you feel the sum operation in the formula may not have been fully comprehended, you can introduce it again in a much simpler context. Say just present that $\sum_{i=1}^{n}(x_i)$ is the same as saying in this example;

  x
 --
  2
  4
  9
  5
+ 0
 --
 20

Now that mess should be cleared up, and we can work our way into the second part of the formula, $(x_i-\bar{x})(y_i-\bar{y})$. Now, assuming people already know what the mean, $\bar{x}$ and $\bar{y}$ stand for, and I would say, being hypocritical of my own comments earlier in the post, one can just refer to the mean in terms of simple heuristics (e.g. the middle of the distribution). One can then just take this process one operation at a time. The statement $(x_i-\bar{x})$ is just examining the deviations/distance between each observation, and the mean of all observations for that particular attribute. Hence when an observation is further from the mean, this operation will be given a higher value. One can then refer back to the example table given, and simply demonstrate the operation on the $x$ vector of observations.

x x_bar (x - x_bar)
2 4     -2
4 4      0
9 4      5
5 4      1
0 4     -4

The operation is the same for $y$ vector, but just for reinforcement you can present that operation as well.

y y_bar (y - y_bar)
5  6     -1
8  6      2
3  6     -3
6  6      0
8  6      2

Now, the terms $(x_i-\bar{x})$ and $(y_i-\bar{y})$ should not be ambiguous, and we can go onto the next operation, multiplying these results together, $(x_i-\bar{x})\cdot(y_i-\bar{y})$. As gung points out in the comments, this is frequently called the cross product (perhaps a useful example to bring back up if one were introducing basic matrix algebra for statistics).

Take note of what happens when multiplying, if two observations are both a large distance above the mean, the resulting observation will have an even larger positive value (the same is true if both observations are a large distance below the mean, as multiplying two negatives equals a positive). Also note that if one observation is high above the mean and the other is well below the mean, the resulting value will be large (in absolute terms) and negative (as a positive times a negative equals a negative number). Finally note that when a value is very near the mean for either observation, multiplying the two values will result in a small number. Again we can just present this operation in a table.

(x - x_bar) (y - y_bar)  (x - x_bar)*(y - y_bar)
-2             -1                2
 0              2                0  
 5             -3              -15 
 1              0                0
-4              2               -8

Now if there are any statisticians in the room they should be boiling with anticipation at this point. We can see all the seperate elements of what a covariance is, and how it is calculated come into play. Now all we have to do is sum up the final result in the preceding table, divide by $n-1$ and voila, the covariance should no longer be mystical (all with only defining one greek symbol).

(x - x_bar)*(y - y_bar)
-----------------------
   2
   0
 -15
   0
+ -8
-----
 -21

-21/(5-1) = -5.25

At this point you may want to reinforce where the 5 is coming from, but that should be as simple as referring back to the table and counting the number of observations (lets again leave the difference between sample and population to another time).

Now, the covariance in and of itself does not tell us much (it can, but it is needless at this point to go into any interesting examples without resorting to magically, undefined references to the audience). In a good case scenario, you won't really need to sell why we should care what the covariance is, in other circumstances, you may just have to hope your audience is captive and will take your word for it. But, continuing on to develop the difference between what the covariance is and what the correlation is, we can just refer back to the formula for correlation. To prevent greek symbol phobia maybe just say $\rho$ is the common symbol used to represent correlation.

$\rho = \frac{Cov(x,y)}{\sqrt{Var(x)Var(y)}}$

Again, to reiterate, the numerator in the preceding formula is simply the covariance as we have just defined, and the denominator is the square root of the product of the variance of each individual series. If you need to define the variance itself, you could just say that the variance is the same thing as the covariance of a series with itself (i.e. $Cov(x,x) = Var(x)$). And all the same concepts that you introduced with the covariance apply (i.e. if a series has many values a far ways from its mean, it will have a high variance). Maybe note here that a series can not have a negative variance as well (which should logically follow from the math previously presented).

So the only new components we have introduced are in the denominator, $Var(x)Var(y)$. So we are dividing the covariance we just calculated by the product of the variances of each series. One could go into the treatment about why dividing by $\sqrt{Var(x)Var(y)}$ will always result in a value between -1 and 1, but I suspect the Cauchy–Schwarz inequality should be left off of the agenda for this discussion. So again, I'm a hypocrite and resort to some, take my word for it, but at this point we can introduce all the reasons why we use the correlation coefficient. One can then relate these math lessons back to the heuristics that have been given in the other statements, such as Peter Flom's response to one of the other questions. While this was critisized for introducing the concept in terms of causal statements, that lesson should be on the agenda at some point as well.

I understand in some circumstances this level of treatment would not be appropriate. The senate needs the executive summary. In that case, well you can refer back to the simple heuristics that people have been using in other examples, but Rome wasn't built in a day. And to the senate whom asks for the executive summary, if you have so little time perhaps you should just take my word for it, and dispense with the formalities of analogies and bullet-points.

Answer 3

Correlation (r) is the covariance (cov) of your variables (x & y) divided by (or adjusted by, in other words) each of their standard deviations ($\sqrt{Var[x]Var[y]}$).

That is, correlation is simply a representation of covariance so the result must lay between -1 (perfectly inversely correlated) an +1 (perfectly positively correlated), noting that a value close to zero means that two variables are uncorrelated.

Covariance is unbounded and lacks a context when comparing to other covariances. By Normalising/adjusting/standardising covariances into a correlation, data sets can be compared more easily.

As you can imagine, there are different ways a statistic (such as covariance) can be normalised/standardised. The mathematical formula for the relationship between correlation and covariance simply reflects the convention statisticians use (namely, adjusting according to their standard deviations):

$$r = \frac{cov(x,y)}{\sqrt{Var[x]Var[y]}}$$

Answer 4

If you are familiar with the idea of centering and standardizing, x-xbar is to center x at its mean. Same applies to y. So covariance simply centers the data. Correlation, however, not only centers the data but also scales using the standard deviation (standardize). The multiplication and summation is the dot-product of the two vectors and it tells how parallel these two vectors compare to each other (the projection of one vector onto the other). The division of (n-1) or taking the expected value is to scale for the number of observations. Thoughts?

Answer 5

As far as I've understood it. Correlation is a "normalized" version of the covariance.

Answer 6

Correlation is scaled to be between -1 and +1 depending on whether there is positive or negative correlation, and is dimensionless. The covariance however, ranges from zero, in the case of two independent variables, to Var(X), in the case where the two sets of data are equal. The units of COV(X,Y) are the units of X times the units of Y.