23
$\begingroup$

First of all, I'm not asking this:

Why does zero correlation not imply independence?

This is addressed (rather nicely) here: https://math.stackexchange.com/questions/444408/why-does-zero-correlation-not-imply-independence

What I'm asking is the opposite...say two variables are entirely independent of one another.

Couldn't they have a tiny bit of correlation by accident?

Shouldn't it be...independence implies VERY LITTLE correlation?

Improve this question
$\endgroup$
6
  • 7
    $\begingroup$ Even independent variables will almost always have a non-zero SAMPLE correlation, though it will likely still be close to zero. $\endgroup$
    – jsk
    Jun 16, 2019 at 22:55
  • 10
    $\begingroup$ As @jsk pointed out, you may be confusing sample correlation with expected correlation $\endgroup$
    – David
    Jun 16, 2019 at 23:00
  • 1
    $\begingroup$ @David could you explain? I'm still very much a beginner in statistics. $\endgroup$
    – joshuaronis
    Jun 17, 2019 at 1:30
  • 3
    $\begingroup$ @JoshuaRonis Sample correlation is the correlation you observe when working with a bunch of data. You use that to get an idea of what the "true" correlation between two variables is. The bigger the sample, the better the estimate you get. For example, the correlation between the results of two dice are independent, therefore uncorrelated, even though if you roll them together ten times, you may get correlation (due to random chance) But please realize that there is no preference for positive nor negative correlation (i.e. you have equal chance of each) $\endgroup$
    – David
    Jun 17, 2019 at 7:17
  • 1
    $\begingroup$ Not a dupe but related discussion: Does non-zero correlation imply dependence? $\endgroup$
    – SecretAgentMan
    Jun 17, 2019 at 14:00

4 Answers 4

Reset to default
42
$\begingroup$

By the definition of the correlation coefficient, if two variables are independent their correlation is zero. So, it couldn't happen to have any correlation by accident!

$$\rho_{X,Y}=\frac{\operatorname{E}[XY]-\operatorname{E}[X]\operatorname{E}[Y]}{\sqrt{\operatorname{E}[X^2]-[\operatorname{E}[X]]^2}~\sqrt{\operatorname{E}[Y^2]- [\operatorname{E}[Y]]^2}}$$

If $X$ and $Y$ are independent, means $\operatorname{E}[XY]= \operatorname{E}[X]\operatorname{E}[Y]$. Hence, the numerator of $\rho_{X,Y}$ is zero in this case.

So, if you don't change the meaning of correlation, as mentioned here, it is not possible. Unless, clarify your defintion from what the correlation is.

Improve this answer
$\endgroup$
6
  • 3
    $\begingroup$ And yet, we have charts clearly showing a (inverse) correlation between number of pirates and global mean temperature . As other comments point out, one must be careful about the sample sizes, not to mention 'accidental appearances' $\endgroup$
    – Carl Witthoft
    Jun 19, 2019 at 15:16
  • $\begingroup$ @OmG "if you don't change the meaning of correlation, as mentioned here" When I read the OPs question, I got a very different meaning of "correlation". To me: "Couldn't they have a tiny bit of correlation by accident?" very strongly implies 'measuring" correlation, and when you measure correlation in reality you will very often find "a tiny bit of correlation by accident". $\endgroup$
    – industry7
    Jun 19, 2019 at 20:13
  • 1
    $\begingroup$ @industry7 I see. But it should be defined in a formal method. It is qualitative and we can't talk about it here. $\endgroup$
    – OmG
    Jun 20, 2019 at 15:00
  • $\begingroup$ @CarlWitthoft The number of pirates and the global mean temperature are not independent. They have a common cause (i.e., time, development, modernization, etc.) that creates a dependence between them. "Independence" doesn't mean "doesn't cause"; it means "unassociated", and clearly those charts demonstrate association. $\endgroup$
    – Noah
    Jul 3, 2019 at 2:59
  • $\begingroup$ @Noah I fear a WHOOSH happened. venganza.org $\endgroup$
    – Carl Witthoft
    Jul 3, 2019 at 12:07
21
$\begingroup$

Comment on sample correlation. In comparing two small independent samples of the same size, the sample correlation is often noticeably different from $r = 0.$ [Nothing here contradicts @OmG's Answer (+1) on the population correlation $\rho.]$

Consider correlations between a million pairs of independent samples of size $n = 5$ from the exponential distribution with rate $1.$

set.seed(616)
r = replicate( 10^6, cor(rexp(5), rexp(5))  )
mean(abs(r) > .5)
[1] 0.386212
mean(r)
[1] -0.0005904455

hist(r, prob=T, br=40, col="skyblue2")
  abline(v=c(-.5,.5), col="red", lwd=2)

enter image description here

For example, here is the scatterplot of first of the million pairs of samples of size $5,$ for which $r = -0.5716.$

enter image description here

There is nothing special about the exponential distribution in this regard. Changing the parent distribution to standard uniform gave the following results.

set.seed(2019)
...
mean(abs(r) > .5)
[1] 0.391061
mean(r)
[1] 1.43269e-05

enter image description here

By contrast, here is the corresponding histogram of correlations for pairs of normal samples of size $n = 20.$

enter image description here

Note: Other pages on this site discuss the distribution of $r$ in more detail; one of them is this Q & A.

Improve this answer
$\endgroup$
1
  • 8
    $\begingroup$ For small sample size, you're likely to find sample correlations that are "noticeably" different from zero, but you're no more likely to find correlations that are significantly different from zero. Even though your point estimate is far from zero, you have far too little data to confidently claim that you're seeing nonzero correlation due to anything but chance. With only 5 pairs, even correlations coefficients greater than 0.8 may not be significantly different from 0. $\endgroup$
    – Nuclear Hoagie
    Jun 17, 2019 at 12:58
16
$\begingroup$

Simple answer: if 2 variables are independent, then the population correlation is zero, whereas the sample correlation will typically be small, but non-zero.

That is because the sample is not a perfect representation of the population.

The larger the sample, the better it represents the population, so the smaller the correlation you'll have. For an infinite sample, the correlation would be zero.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ The precise formulation would be that for any $p$ and $\epsilon$, there is some $n$ such that if the sample size is greater than $n$, then the probability of the correlation being greater than $\epsilon$ is less than $p$. $\endgroup$
    – Acccumulation
    Jun 17, 2019 at 15:06
  • $\begingroup$ Yes, absolutely correct! I tried to keep my answer as simple and conceptual as possible. $\endgroup$
    – Dave
    Jun 17, 2019 at 16:24
1
$\begingroup$

Maybe this is helpful for some people sharing the same intuitive understanding. We've all seen something like this:

enter image description here

These data are presumably independent but clearly exhibit correlation ($r = 0.66$). "I thought independence implies zero correlation!" the student says.

As others have already pointed out, the sample values are correlated, but that does not mean the population has nonzero correlation.

Of course, these two should be independent—given Nicolas Cage appeared in a record-setting 10 films this year, we shouldn't be closing the local pool for the summer for safety purposes.

But when we check how many people drown this year, there is a small chance that a record-setting 1000 people drown this year.

Getting such correlation is unlikely. Maybe one in a thousand. But it's possible, even though the two are independent. But this is just one case. Consider that there the millions of possible events to measure out there, and you can see the chance that the odds of some two happening to give a high correlation is quite high (hence the existence of graphs such as that above).

Another way to look at it is that guaranteeing that two independent events will always give uncorrelated values is itself restrictive. Given two independent dice, and the results of the first, there are a certain (sizable) set of results for the second dice which will give some nonzero correlation. To restrict the second dice's results to give zero correlation with the first is a clear violation of independence, as the first dice's rolls are now affecting the distribution of the results.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.