What is covariance in plain language and how is it linked to the terms dependence, correlation and variance-covariance structure with respect to repeated-measures designs?


Covariance is a measure of how changes in one variable are associated with changes in a second variable. Specifically, covariance measures the degree to which two variables are linearly associated. However, it is also often used informally as a general measure of how monotonically related two variables are. There are many useful intuitive explanations of covariance here.

Regarding how covariance is related to each of the terms you mentioned:

(1) Correlation is a scaled version of covariance that takes on values in $[-1,1]$ with a correlation of $\pm 1$ indicating perfect linear association and $0$ indicating no linear relationship. This scaling makes correlation invariant to changes in scale of the original variables, (which Akavall points out and gives an example of, +1). The scaling constant is the product of the standard deviations of the two variables.

(2) If two variables are independent, their covariance is $0$. But, having a covariance of $0$ does not imply the variables are independent. This figure (from Wikipedia)

$ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ $ enter image description here

shows several example plots of data that are not independent, but their covariances are $0$. One important special case is that if two variables are jointly normally distributed, then they are independent if and only if they are uncorrelated. Another special case is that pairs of bernoulli variables are uncorrelated if and only if they are independent (thanks @cardinal).

(3) The variance/covariance structure (often called simply the covariance structure) in repeated measures designs refers to the structure used to model the fact that repeated measurements on individuals are potentially correlated (and therefore are dependent) - this is done by modeling the entries in the covariance matrix of the repeated measurements. One example is the exchangeable correlation structure with constant variance which specifies that each repeated measurement has the same variance, and all pairs of measurements are equally correlated. A better choice may be to specify a covariance structure that requires two measurements taken farther apart in time to be less correlated (e.g. an autoregressive model). Note that the term covariance structure arises more generally in many kinds of multivariate analyses where observations are allowed to be correlated.

  • 2
    $\begingroup$ your explanation is nice. It is followed by valuable supplement which caused an interesting series of comments. Thanks a lot to all :) ! $\endgroup$ – stan Jun 7 '12 at 7:23

Macro's answer is excellent, but I want to add more to a point of how covariance is related to correlation. Covariance doesn't really tell you about the strength of the relationship between the two variables, while correlation does. For example:

x = [1, 2, 3]
y = [4, 6, 10]

cov(x,y) = 2 #I am using population covariance here

Now let's change the scale, and multiply both x and y by 10

x = [10, 20, 30]
y = [40, 60, 100]

cov(x, y) = 200

Changing the scale should not increase the strength of the relationship, so we can adjust by dividing the covariances by standard deviations of x and y, which is exactly the definition of correlation coefficient.

In both above cases correlation coefficient between x and y is 0.98198.

  • 6
    $\begingroup$ "Covariance doesn't really tell you about the strength of the relationship between the two variables, while correlation does." That statement is completely false. The two measures are identical modulo scaling by the two standard deviations. $\endgroup$ – David Heffernan Jun 3 '12 at 15:35
  • 13
    $\begingroup$ @DavidHeffernan, yes if scaled by standard deviations then covariance tells us about the strength of the relationship. However, covariance measure by it self doesn't tell us that. $\endgroup$ – Akavall Jun 3 '12 at 16:48
  • 8
    $\begingroup$ @DavidHeffernan, I think what Akavall is saying is that if you don't know the scale of the variables then covariance does not tell you anything about the strength of the relationship - only the sign can be interpreted. $\endgroup$ – Macro Jun 3 '12 at 16:51
  • 6
    $\begingroup$ In what practical situation can you obtain a covariance without also being able to obtain a good estimate of the scale of the variables? $\endgroup$ – David Heffernan Jun 3 '12 at 18:45
  • 7
    $\begingroup$ However, it is not always necessary to know the standard deviation to understand the scale of a variable and thus the strength of a relationship. Unstandardised effects are often informative. E.g., if doing a training course causes people to on average increase there income by $10,000 per year, that's probably a better indication of strength of effect, than saying that there was a r=.34 correlation between doing the course and income. $\endgroup$ – Jeromy Anglim Jun 4 '12 at 1:39

protected by whuber Aug 29 '12 at 13:11

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.