13
$\begingroup$

I am getting some perplexing results for the correlation of a sum with a third variable when the two predictors are negatively correlated. What is causing these perplexing results?

Example 1: Correlation between the sum of two variables and a third variable

Consider formula 16.23 on page 427 of Guildford's 1965 text, shown below.

Perplexing finding: If both variables correlate .2 with the third variable and correlate -.7 with each other, the formula results in a value of .52. How can the correlation of the total with the third variable be .52 if the two variables each correlate only .2 with the third variable?

Example 2: What is the multiple correlation between two variables and a third variable?

Consider formula 16.1 on page 404 of Guildford's 1965 text (shown below).

Perplexing finding: Same situation. If both variables correlate .2 with the third variable and correlate -.7 with each other, the formula results in a value of .52. How can the correlation of the total with the third variable be .52 if the two variables each correlate only .2 with the third variable?

I tried a quick little Monte Carlo simulation and it confirms the results of the Guilford formulas.

But if the two predictors each predict 4% of the variance of the third variable, how can a sum of them predict 1/4 of the variance?

correlation of sum of two variables with a third variable multiple correlation of two variables with a third variable

Source: Fundamental Statistics in Psychology and Education, 4th ed., 1965.

CLARIFICATION

The situation I am dealing with involves predicting future performance of individual people based on measuring their abilities now.

The two Venn diagrams below show my understanding of the situation and are meant to clarify my puzzlement.

This Venn diagram (Fig 1) reflects the zero order r=.2 between x1 and C. In my field there are many such predictor variables that modestly predict a criterion.

Fig. 1

This Venn diagram (Fig 2) reflects two such predictors, x1 and x2, each predicting C at r=.2 and the two predictors negatively correlated, r=-.7.

Fig. 2

I am at a loss to envision a relationship between the two r=.2 predictors that would have them together predict 25% of the variance of C.

I seek help understanding the relationship between x1, x2, and C.

If (as suggested by some in reply to my question) x2 acts as a suppressor variable for x1, what area in the second Venn diagram is being suppressed?

If a concrete example would be helpful, we can consider x1 and x2 to be two human ability and C to be 4 year college GPA, 4 years later.

I am having trouble envisioning how a suppressor variable could cause the 8% explained variance of the two r=.2 zero order r's to enlarge and explain 25% of the variance of C. A concrete example would be a very helpful answer.

$\endgroup$
  • $\begingroup$ There's an old rule of thumb in statistics that the variance of the sum of a set of independent variables is equal to the sum of their variances. $\endgroup$ – Mike Hunter Jan 20 '17 at 0:45
  • $\begingroup$ @DJohnson. How does your comment relate to the question asked? $\endgroup$ – Joel W. Jan 20 '17 at 14:19
  • $\begingroup$ Sorry, I don't understand the question. To me, it's obvious how it relates. Besides, it's a comment that's neither eligible for the bounty nor requiring deeper elaboration. $\endgroup$ – Mike Hunter Jan 20 '17 at 14:40
  • 1
    $\begingroup$ @DJohnson. How does your comment relate to the question asked? To me, it is NOT obvious how it relates. $\endgroup$ – Joel W. Jan 20 '17 at 15:28
  • 2
    $\begingroup$ Your question about the meaning of N views might get a better response on the Meta CV site. $\endgroup$ – mdewey Jan 20 '17 at 17:38
3
+100
$\begingroup$

This can happen when the two predictors both contain a large nuisance factor, but with opposite sign, so when you add them up the nuisance cancels out and you get something much closer to the third variable.

Let's illustrate with an even more extreme example. Suppose $X, Y \sim N(0,1)$ are independent standard normal random variables. Now let

$A = X$

$B = -X + 0.00001Y$

Say that $Y$ happens to be your third variable, $A, B$ are your two predictors, and $X$ is a latent variable you don't know anything about. The correlation of A with Y is 0, and the correlation of B with Y is very small, close to 0.00001.* But the correlation of $A+B$ with $Y$ is 1.

*There is a teeny tiny correction for the standard deviation of B being a bit more than 1.

$\endgroup$
  • $\begingroup$ Does this type of situation ever arise in the social sciences? $\endgroup$ – Joel W. Jan 22 '17 at 2:59
  • 1
    $\begingroup$ In social science jargon, this is basically just a strong effect confounding a weak effect in a particular way. I'm not a social science expert, but I can't imagine it's hard to find an example of that. $\endgroup$ – Paul Jan 22 '17 at 3:02
  • $\begingroup$ Might you have any examples from other than the physical sciences? $\endgroup$ – Joel W. Jan 22 '17 at 3:05
  • $\begingroup$ Can the relationship you describe be shown in a Venn diagram? $\endgroup$ – Joel W. Jan 22 '17 at 3:06
  • $\begingroup$ I wouldn't personally find a Venn diagram helpful here but if you must, I would draw B as a rectangle, then split it into two sub-rectangles, a big fat one A and a tiny skinny one Y. Summing A and B is canceling out the big part A and leaving the tiny part Y. $\endgroup$ – Paul Jan 22 '17 at 3:19
10
$\begingroup$

It can be helpful to conceive of the three variables as being linear combinations of other uncorrelated variables. To improve our insight we may depict them geometrically, work with them algebraically, and provide statistical descriptions as we please.

Consider, then, three uncorrelated zero-mean, unit-variance variables $X$, $Y$, and $Z$. From these construct the following:

$$U = X,\quad V = (- 7 X + \sqrt{51}Y )/10;\quad W=(\sqrt{3} X + \sqrt{17} Y + \sqrt{55}Z)/\sqrt{75}.$$

Geometric Explanation

The following graphic is about all you need in order to understand the relationships among these variables.

Figure

This pseudo-3D diagram shows $U$, $V$, $W$, and $U+V$ in the $X,Y,Z$ coordinate system. The angles between the vectors reflect their correlations (the correlation coefficients are the cosines of the angles). The large negative correlation between $U$ and $V$ is reflected in the obtuse angle between them. The small positive correlations of $U$ and $V$ with $W$ are reflected by their near-perpendicularity. However, the sum of $U$ and $V$ fall directly beneath $W$, making an acute angle (around 45 degrees): there's the unexpectedly high positive correlation.


Algebraic Calculations

For those wanting more rigor, here is the algebra to back up the geometry in the graphic.

All those square roots are in there to make $U$, $V$, and $W$ have unit variances, too: that makes it easy to compute their correlations, because the correlations will equal the covariances. Therefore

$$\operatorname{Cor}(U, V) = \operatorname{Cov}(U,V) = \mathbb{E}(UV) = \mathbb{E}(\sqrt{51}XY- 7 X^2)/10 = -7/10 = -0.7$$

because $X$ and $Y$ are uncorrelated. Similarly,

$$\operatorname{Cor}(U,W) = \sqrt{3/75} = 1/5 = 0.2$$

and

$$\operatorname{Cor}(V,W) = (-7\sqrt{3} + \sqrt{15}\sqrt{17})/(10\sqrt{75}) = 1/5 = 0.2.$$

Finally,

$$\operatorname{Cor}(U+V,W) = \frac{\operatorname{Cov}(U+V,W)}{\sqrt{\operatorname{Var}(U+V)\operatorname{Var}(W)}} = \frac{1/5 + 1/5}{\sqrt{\operatorname{Var}(U) + \operatorname{Var}(V) + 2\operatorname{Cov}(U,V)}} = \frac{2/5}{\sqrt{1 + 1 - 2(7/10)}} = \frac{2/5}{\sqrt{3/5}}\approx 0.5164.$$

Consequently these three variables do have the desired correlations.


Statistical Explanation

Now we can see why everything works out as it does:

  • $U$ and $V$ have a strong negative correlation of $-7/10$ because $V$ is proportional to the negative of $U$ plus a little "noise" in the form of a small multiple of $Y$.

  • $U$ and $W$ have weak positive correlation of $1/5$ because $W$ includes a small multiple of $U$ plus a lot of noise in the form of multiples of $Y$ and $Z$.

  • $V$ and $W$ have weak positive correlation of $1/5$ because $W$ (when multiplied by $\sqrt{75}$, which won't change any correlations) is the sum of three things:

    • $\sqrt{17}Y$, which is positively correlated with $V$;
    • $-\sqrt{3}X$, whose negative correlation with $V$ reduces the overall correlation;
    • and a multiple of $Z$ which introduces a lot of noise.
  • Nevertheless, $U+V = (3X + \sqrt{51}Y)/10 = \sqrt{3/100}(\sqrt{3}X + \sqrt{17}Y)$ is rather positively correlated with $W$ because it is a multiple of that part of $W$ which does not include $Z$.

$\endgroup$
  • $\begingroup$ Is there a way to show this in a Venn diagram? Despite the math, I still do not see the logic of the sum of two variables explaining 25+% of the variance of a third variable when each off the two variables that go into the sum predict but 4% of the variance of that third variable. How can 8% explained variance become 25% explained variance just by adding the two variables? $\endgroup$ – Joel W. Jan 15 '17 at 0:18
  • $\begingroup$ Also, are there practical applications of this strange phenomenon? $\endgroup$ – Joel W. Jan 15 '17 at 2:06
  • $\begingroup$ If a Venn diagram is inappropriate to represent explained variance, can you tell me why it is inappropriate? $\endgroup$ – Joel W. Jan 17 '17 at 0:43
  • 1
    $\begingroup$ @JoelW. The nice answer here touches on why Venn diagrams are not up to the task of illustrating this phenomenon (toward the end of the answer): stats.stackexchange.com/a/73876/5829 $\endgroup$ – Jake Westfall Jan 19 '17 at 23:44
  • $\begingroup$ Joel, the Cohens used a Venn-like diagram they called a "Ballantine" for analyzing variances. See ww2.amstat.org/publications/jse/v10n1/kennedy.html for instance. As far as practical applications go, you ought to be asking the opposite question: what applications of variance and variance decompositions are not practical? $\endgroup$ – whuber Jan 20 '17 at 0:10
5
$\begingroup$

Another simple example:

  • Let $z \sim \mathcal{N}(0,1)$
  • Let $x_1 \sim \mathcal{N}(0,1)$
  • Let $x_2 = z - x_1$ (hence $z = x_1 + x_2$)

Then:

  • $\mathrm{Corr}(z, x_1) = 0$
  • $\mathrm{Corr}(z, x_2) \approx .7$
  • $\mathrm{Corr}(z, x_1 + x_2) = 1$

Geometrically, what's going on is like in WHuber's graphic. Conceptually, it might look something like this: enter image description here

(At some point in your math career, it can be enlightening to learn that random variables are vectors, $E[XY]$ is an inner product, and hence correlation is the cosine of the angle between the two random variables.)

$x_1$ and $z$ are uncorrelated, hence they're orthogonal. Let $\theta$ denote the angle between two vectors.

  • $\mathrm{Corr}(z, x_1) = \cos \theta_{zx_1} = 0 \quad \quad \theta_{z,x_1} = \frac{\pi}{2}$
  • $\mathrm{Corr}(z, x_2) = \cos \theta_{zx_2} \approx .7 \quad \quad \theta_{z,x_2} = \frac{\pi}{4} $
  • $\mathrm{Corr}(z, x_1 + x_2) = \cos \theta_{z,x_1+x_2} = 1 \quad \quad \theta_{z, x_1 + x_2} = 0$

To connect to the discussion in the comments Flounderer's answer, think of $z$ as some signal, $-x_1$ as some noise, and noisy signal $x_2$ as the sum of signal $z$ and noise $-x_1$. Adding $x_1$ to $x_2$ is equivalent to subtracting noise $-x_1$ from the noisy signal $x_2$.

$\endgroup$
  • $\begingroup$ (+1) Nice example! $\endgroup$ – user795305 Jan 20 '17 at 5:56
  • $\begingroup$ Please explain the premises of your answer. After positing z = x1 + x2, why say “then Corr(z,x1)=0”? Are you saying that Corr(z,x1)=0 follows from your first Let statement, or is the correlation of zero an additional assumption? If it is an additional assumption, why does the situation in the original question require that additional assumption? $\endgroup$ – Joel W. Jan 20 '17 at 17:02
  • $\begingroup$ @JoelW. I'm saying $z$ is a random variable following the standard normal distribution and $x_1$ is an independent random variable that also follows the standard normal distribution. $z$ and $x_1$ are independent, hence their correlation is precisely 0. Then compute $z - x_1$ and call that $x_2$. $\endgroup$ – Matthew Gunn Jan 20 '17 at 17:05
  • $\begingroup$ @MatthewGunn. Your third Let says z=x1+x2. That seems to violate your first two Lets that say that z and x1 are independent. $\endgroup$ – Joel W. Jan 20 '17 at 17:23
  • 1
    $\begingroup$ @JoelW. I do not agree because that statement is not true. Seeing $z = x_1 + x_2$ implies nothing about independence between $z$ and $x_1$. $\endgroup$ – Matthew Gunn Jan 20 '17 at 21:39
3
$\begingroup$

Addressing your comment:

Despite the math, I still do not see the logic of the sum of two variables explaining 25+% of the variance of a third variable when each off the two variables that go into the sum predict but 4% of the variance of that third variable. How can 8% explained variance become 25% explained variance just by adding the two variables?

The issue here seems to be the terminology "variance explained". Like a lot of terms in statistics, this has been chosen to make it sound like it means more than it really does.

Here's a simple numerical example. Suppose some variable $Y$ has the values

$$y = (6, 7, 4, 8, 9, 6, 6, 3, 5, 10)$$

and $U$ is a small multiple of $Y$ plus some error $R$. Let's say the values of $R$ are much larger than the values of $Y$.

$$r = (-20, -80, 100, 90, 50, 70, 40, 30, 40, 60)$$

and $U = R + 0.1Y$, so that

$$u = (-19.4, -79.3, 100.4, 90.8, 50.9, 70.6, 40.6, 30.3, 40.5, 61.0)$$

and suppose another variable $V=-R+0.1Y$ so that

$$v = (20.6, 80.7, -99.6, -89.2, -49.1, -69.4, -39.4, -29.7, -39.5, -59.0)$$

Then both $U$ and $V$ have very small correlation with $Y$, but if you add them together then the $r$'s cancel and you get exactly $0.2Y$, which is perfectly correlated with $Y$.

In terms of variance explained, this makes perfect sense. $Y$ explains a very small proportion of the variance in $U$ because most of the variance in $U$ is due to $R$. Similarly, most of the variance in $V$ is due to $R$. But $Y$ explains all of the variance in $U+V$. Here is a plot of each variable:

Plot of each of the variables

However, when you try to use the term "variance explained" in the other direction, it becomes confusing. This is because saying that something "explains" something else is a one-way relationship (with a strong hint of causation). In everyday language, $A$ can explain $B$ without $B$ explaining $A$. Textbook authors seem to have borrowed the term "explain" to talk about correlation, in the hope that people won't realise that sharing a variance component isn't really the same as "explaining".

$\endgroup$
  • $\begingroup$ @naught101 has created some figures to illustrate your variables, Flounderer. You might want to see if including them appeals to you. $\endgroup$ – gung - Reinstate Monica Jan 16 '17 at 1:10
  • $\begingroup$ Sure, edit it however you like. I can't actually view imgur at work but I'm sure it will be fine! $\endgroup$ – Flounderer Jan 16 '17 at 1:11
  • $\begingroup$ I rejected the suggestion, b/c I didn't see that he had contacted you here. You can approve it by going to the suggested edit queue, though. $\endgroup$ – gung - Reinstate Monica Jan 16 '17 at 1:12
  • $\begingroup$ The example you provide is interesting, if carefully crafted, but the situation I presented is more general (with the numbers not carefully chosen) and based on 2 variables N(0,1). Even if we change the terminology from "explains" to "shared", the question remains. How can 2 random variables, each with 4% shared variance with a third variable, be combined in terms of a simple sum that, according to the formula, has 25% shared variance with a third variable? Also, if the goal is prediction, are there any real-world practical applications of this strange increase in shared variance? $\endgroup$ – Joel W. Jan 16 '17 at 2:06
  • $\begingroup$ Well, anywhere in electronics when you have (loud noise + weak signal) + (-loud noise) = weak signal, you would be applying this. For example, noise-cancelling headphones. $\endgroup$ – Flounderer Jan 16 '17 at 2:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.