9
$\begingroup$

I am reading the book Introduction to Statistical Learning and on page 183, the book states that

Since the mean of many highly correlated quantities has higher variance than does the mean of many quantities that are not as highly correlated, the test error estimate resulting from LOOCV tends to have higher variance than does the test error estimate resulting from k-fold CV.

I found a formula that says

Var(𝑋+𝑌)=Var(𝑋)+Var(𝑌)+2Cov(𝑋,𝑌)

which kind of explains the above statement but I still have a hard time understanding it intuitively. Can anyone ELI5 this in layman's term?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ This is not an answer to your question, but you may find the following question/answer interesting since it bears on what you're reading about: stats.stackexchange.com/questions/280665/… $\endgroup$
    – Jake Westfall
    Commented May 3, 2020 at 16:47
  • $\begingroup$ @JakeWestfall Thanks, Jake! That was indeed an interesting read for me. $\endgroup$
    – Dat Nguyen
    Commented May 4, 2020 at 0:18

4 Answers 4

Reset to default
10
$\begingroup$

Say you have a normal six-faced die. And you are interested in the mean of the two numbers you get after rolling it twice.

Scenario 1: You roll the die twice and you get {5} and {3}. Their total is 8 and their mean is 4, while we know the expected value is 3.5. We roll again and we get {2} and {5}, their mean is 3.5. We got quite close to the true expected value.

Scenario 2: You roll the die once, and then you roll the die until you get a number that is at most $\pm$1 away from your first roll. I roll a {6}, hence I can only get a {5} or a {6}. Their mean will be 5.5 or 6. After getting one of those I roll again, I get a {3}. The second roll is a {2}, their mean is 2.5.

In Scenario 1 the two rolling of the die are independent and uncorrelated, hence they can freely explore the sample space. In Scenario 2 the two values are highly correlated, and the sample space is constrained for the second roll, hence it is easier to get more extreme sample means (like 1.5 or 5.5) more often.

We also note that for Scenario 1 there are many ways you can get the same sample mean that corresponds to the true mean: {1} and {6}, {5} and {2}, {4} and {3}. Whereas in Scenario 2 only {3} and {4} will give you the true population mean, as such, the sample means are more variable in the latter case.

Edit for negative covariance:

Consider now a Scenario 3, which is similar to Scenario 2 in that the second roll is also constrained, but in this case the rule for the second roll is a little bit more tricky: if our first roll is below 3.5 (the expected value), we will only accept rolls that are at least $+$3 away from the first value, and if it is above 3.5, we will only accept rolls that are at least $-$3 away from the first value. We roll once and we get a {4}, the only value we can accept then will be a {1}, giving us a sample mean of 2.5. We roll again and we get a {2}, leaving us as possible values for the second roll only {5} and {6}. The sample mean will be 3.5 or 4.

We can see that the sample space is constrained for both Scenario 2 and Scenario 3, but while the first constrains the space so that it is more likely to get extreme sample means—like {1} and {2}—the latter constrains the space so that it is more unlikely to get extreme sample means—it is not possible to get {1} and {2} anymore, nor {1} and {3}. As such, the possible sample means are less variable and closer to the true expected value. This is the effect of a high negative covariance, so the sign is relevant in interpreting the original statement.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you, I get it now! This is the answer for non-stats people like me are looking for. $\endgroup$
    – Dat Nguyen
    Commented May 4, 2020 at 0:08
  • 2
    $\begingroup$ How does this play out in a scenario where $cov(X,Y)<0$? $\endgroup$
    – Dave
    Commented May 4, 2020 at 14:29
  • 1
    $\begingroup$ @Dave edited to include that scenario. Perhaps there is a relationship with regression to the mean that could be mentioned but I think that'd stray away from the simplicity of the example. $\endgroup$
    – Kuku
    Commented May 4, 2020 at 18:16
7
$\begingroup$

The image below might give an intuitive view

without words

This image also shows that high correlation does not always mean higher variance, or is at least ambiguous (that is, the image on the left has a high negative correlation and the result is a low variance for the sum $x+y$).

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ I think the image might lend itself to confusion as the first figure is highly correlated but shows a lower variance than the uncorrelated case, hence having to clarify that "high" does not imply its absolute value. $\endgroup$
    – Kuku
    Commented May 4, 2020 at 14:12
6
$\begingroup$

An extreme example to complement the other answer: making $N$ exact copies of one sample gives me $N$ completely correlated samples. Clearly, this does not reduce the variance of any estimates made using the samples.

We can show this with your formula making two copies $$ \text{Var}(\bar x) = \text{Var}\left(\frac{x+x}{2}\right) = \frac14 \left[\text{Var}(x) + \text{Var}(x) + 2\,\text{Cov}(x,x)\right] = \text{Var}(x) $$ The result can be seen by making $(x+x)/2 = x$ or by recognising that $\text{Cov}(x,x) = \text{Var}(x)$.

Improve this answer
$\endgroup$
1
1
$\begingroup$

Because "highly correlated" generally means Cov(X,Y) is +ve and "uncorrelated" means Cov(X,Y) is zero, so comparing "highly correlated" with "uncorrelated" using your expression you would have Var(𝑋+𝑌) highest in the "highly correlated" case (Var(X) and Var(Y) are always positive.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.