16
$\begingroup$

Can somebody illustrate, as Greg does, but in more detail, how random variables can be dependent, but have zero covariance? Greg, a poster here, gives an example using a circle here.

Can somebody explain this process in more detail using a sequence of steps that illustrate the process at several stages?

Also, if you know of an example from psychology, please illustrate with this concept with a related example. Please be very precise and sequential in your explanation, and also state what some of the consequences might be.

Improve this question
$\endgroup$
3

4 Answers 4

Reset to default
26
$\begingroup$

The basic idea here is that covariance only measures one particular type of dependence, therefore the two are not equivalent. Specifically,

  • Covariance is a measure how linearly related two variables are. If two variables are non-linearly related, this will not be reflected in the covariance. A more detailed description can be found here.

  • Dependence between random variables refers to any type of relationship between the two that causes them to act differently "together" than they do "by themselves". Specifically, dependence between random variables subsumes any relationship between the two that causes their joint distribution to not be the product of their marginal distributions. This includes linear relationships as well as many many others.

  • If two variables are non-linearly related, then they can potentially have 0 covariance but are still dependent - many examples are given here and this plot below from wikipedia gives some graphical examples in the bottom row:

    enter image description here

  • One example where zero covariance and independence between random variables are equivalent conditions is when the variables are jointly normally distributed (that is, the two variables follow a bivariate normal distribution, which is not equivalent to the two variables being individually normally distributed). Another special case is that pairs of bernoulli variables are uncorrelated if and only if they are independent (thanks @cardinal). But, in general the two cannot be taken to be equivalent.

Therefore, one cannot, in general, conclude that two variables are independent just because they appear uncorrelated (e.g. didn't fail to reject the null hypothesis of no correlation). One is well advised to plot data to infer whether the two are related, not just stopping at a test of correlation. For example, (thanks @gung), if one were to run a linear regression (i.e. testing for non-zero correlation) and found a non-sigificant result, one may be tempted to conclude that the variables are not related, but you've only investigated a linear relationship.

I don't know much about Psychology but it makes sense that there could be non-linear relationships between variables there. As a toy example, it seems possible that cognitive ability is non-linearly related to age - very young and very old people are not as sharp as a 30 year old. If one were to plot some measure of cognitive ablity vs. age one may expect to see that cognitive ability is highest at a moderate age and decays around that, which would be a non-linear pattern.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Just a side (pedantic?!) note, but Bernoulli random variables are independent if and only if they are uncorrelated. :) $\endgroup$
    – cardinal
    Jun 10, 2012 at 14:42
  • $\begingroup$ @cardinal, don't worry that's just me abandoning rationality again, kind of like when you said that a multivariate normal with a singular covariance matrix was "commonly used and statistically relevant". $\endgroup$
    – Macro
    Jun 10, 2012 at 14:43
  • $\begingroup$ The next time I'm in Ann Arbor, I'll buy you a coffee to try to offset that joke. :) Feel free to question my rationality in the meantime. :) $\endgroup$
    – cardinal
    Jun 10, 2012 at 14:49
  • $\begingroup$ Ah, but that last citation is true. ;-) It shows up in some surprisingly common places. :) (Though that's getting a little off-topic here.) $\endgroup$
    – cardinal
    Jun 10, 2012 at 14:49
  • $\begingroup$ (+1) I have been kind of on the fence about whether this question should be closed as a duplicate or not. But, I think good answers can make very similar questions worth keeping. Having everything cross-linked helps. $\endgroup$
    – cardinal
    Jun 10, 2012 at 15:45
7
$\begingroup$

A standard way of teaching/visualizing a correlation or covariance is to plot the data, draw lines at the mean of 'x' and 'y', then draw rectangles from the point of the 2 means to the individual datapoints, like this:

enter image description here

The rectangles (points) in the top right and bottom left quadrants (red in the example) contribute positive values to the correlation/covariance, while the rectangles (points) in the top left and bottom right quadrants (blue in the example) contribute negative values to the correlation/covariance. If the total area of the red rectangles equals the total area of the blue rectangles then the positives and negatives cancel out and you get a zero covariance. If there is more area in the red then the covariance will be positive and if there is more area in the blue then the covariance will be negative.

Now lets look at an example from the previous discussion:

enter image description here

The individual points follow a parabola, so they are dependent, if you know 'x' then you know 'y' exactly, but you can also see that for every red rectangle there is a matching blue rectangle, so the final covariance will be 0.

Improve this answer
$\endgroup$
3
  • $\begingroup$ (+1) is there an R package that makes these plots (I recall whuber displaying a plot like this once) or did you do this from scratch? $\endgroup$
    – Macro
    Jun 11, 2012 at 17:34
  • $\begingroup$ @Macro, good question, though I think whuber's were done in Mathematica. It is straightforward to do this "by hand" in R using polygon or rect and a device that supports alpha transparency. $\endgroup$
    – cardinal
    Jun 11, 2012 at 17:43
  • $\begingroup$ I wrote a function to do this plot and will probably add it to the TeachingDemos package sometime soon. My first thought was to shorten the phrase "correlation rectangles" to "correct" as the name of the function, then after a bit realized that name may be easily misunderstood as doing something quite different. So I need to come up with a better name, add a couple options and upload it to R-Forge. $\endgroup$
    – Greg Snow
    Jun 11, 2012 at 21:14
3
$\begingroup$

One simple test if that if the data are basically following a pattern that symmetrical around a vertical or horizontal axis through the means, the co-variance will be pretty close to zero. For example, if the symmetry is around the y-axis, it means that for each value with a given y, there is a positive x difference from mean x and a negative difference from the mean x. The addition of y*x for those values will be zero. You can see this illustrated nicely in the collection of example plots in the other answers. There are other patterns that would yield a zero co-variance but not independence, but many examples are easily evaluated by looking for symmetry or not.

Improve this answer
$\endgroup$
2
$\begingroup$

An example from Wikipedia:

"If the variables are independent, Pearson's correlation coefficient is 0, but the converse is not true because the correlation coefficient detects only linear dependencies between two variables. For example, suppose the random variable X is symmetrically distributed about zero, and Y = X^2. Then Y is completely determined by X, so that X and Y are perfectly dependent, but their correlation is zero; they are uncorrelated. However, in the special case when X and Y are jointly normal, uncorrelatedness is equivalent to independence."

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.