4
$\begingroup$

Pretty much what it says in the title. I don't know too much about statistics and I worry I'm getting this wrong.

There are variables $X$ and $Y$, they are uncorrelated by design, because one has been corrected for the other. We want to find out how much of the variance in $Z$ is explained by $X$ and $Y$. $X$ explains 40% of the variance of $Z$, i.e. the $r^2 = 0.40$. $Y$ explains 50% of the variance in $Z$.

Usually you can't add these percentages to get the variance in $Z$ explained by $X$ and $Y$, because it might be the case that $X$ does not explain any variance in $Z$ beyond what is already explained by $Y$.

But in that case $X$ and $Y$ should be strongly correlated. So is the absence of a correlation between $X$ and $Y$ enough to allow us to add the percentages of variance explained or is there something that I'm missing?

$\endgroup$
3
  • $\begingroup$ What do you mean by "$X$ and $Y$ are uncorrelated by design, because one has been corrected for the other"? $\endgroup$
    – Ale
    Jun 8 '20 at 13:15
  • 1
    $\begingroup$ It means $Y$ is a residual of a linear regression of $Y'$ on $X$, but that's not really important for the question. $\endgroup$ Jun 8 '20 at 13:20
  • $\begingroup$ It is important. It sounds like you might be asking an XY problem where your real issue has to do with residuals, and your attempt to solve that issue has led you to wonder about uncorrelated features: en.m.wikipedia.org/wiki/XY_problem. $\endgroup$
    – Dave
    Jun 8 '20 at 13:48
2
$\begingroup$

Yes (as per the derivation below - there are still things I am not clear about, e.g., what if each regressor explains more than 50% of the variance: can we then show they cannot be uncorrelated anymore, as $R^2$ can evidently not add up to more than one? I did play around with higher $R^2$s such as resulting from y <- 4*x1.c + 5*x2 + rnorm(n, sd=.01) below, though, which suggests the problem does not occur. See also Thomas Lumley's helpful comment below!).

Also, the derivation demonstrates that the result goes through for general multiple regressions.

Let us suppose your regressions of $y$ on $X_1$ or $X_2$ or both contain a constant (if they do not, a similar result could be established for uncentered $R^2$).

By the FWL theorem, that is equivalent to regressing demeaned dependent variables on demeaned regressors.

Call these $\tilde y=y-\bar{y}$, $\tilde X_1=X_1-\bar{X}_1$ and $\tilde X_2=X_2-\bar{X}_2$, with suitably defined $\bar{X}_j$ for the column means of the regressor matrices. In the absence of correlation, we then have $\tilde X_1'\tilde X_2=0$.

The formula for $R^2$ is $$ R^{2}:=1-\frac{e'e}{\tilde{y}'\tilde{y}} $$ for a residual vector $e$. When regressing $\tilde y$ on, e.g., $\tilde X_1$, we have $e=M_{\tilde X_1}\tilde y$, with $M_{\tilde X_1}$ the usual residual maker matrix.

Hence, when adding the $R^2$s of the separate regressions, we have $$ R_1^2+R_2^2=1-\frac{\tilde y'M_{\tilde X_1}\tilde y}{\tilde{y}'\tilde{y}}+1-\frac{\tilde y'M_{\tilde X_2}\tilde y}{\tilde{y}'\tilde{y}}=\frac{2\tilde y'\tilde y-\tilde y'M_{\tilde X_1}\tilde y-\tilde y'M_{\tilde X_2}\tilde y}{\tilde y'\tilde y} $$ Using $\tilde y'M_{\tilde X_j}\tilde y=\tilde y'\tilde y-\tilde y'P_{\tilde X_j}\tilde y$, $j=1,2$, we obtain $$ R_1^2+R_2^2=\frac{\tilde y'P_{\tilde X_1}\tilde y+\tilde y'P_{\tilde X_2}\tilde y}{\tilde y'\tilde y}, $$ with $P$ the projection matrices.

Next, consider the estimator of the joint regression of $\tilde{y}$ on both $\tilde{X}_1$ and $\tilde{X}_2$ when regressors are orthogonal: $$ \begin{eqnarray*} b&=&\left(% \begin{array}{cc} \tilde{X}_1'\tilde{X}_1 & 0 \\ 0 & \tilde{X}_2'\tilde{X}_2 \\ \end{array}% \right)^{-1}\left(% \begin{array}{c} \tilde{X}_1'\tilde{y} \\ \tilde{X}_2'\tilde{y} \\ \end{array}% \right)\\ &=& \left(% \begin{array}{cc} (\tilde{X}_1'\tilde{X}_1)^{-1} & 0 \\ 0 & (\tilde{X}_2'\tilde{X}_2)^{-1} \\ \end{array}% \right)\left(% \begin{array}{c} \tilde{X}_1'\tilde{y} \\ \tilde{X}_2'\tilde{y} \\ \end{array}% \right)\\ &=&\left(% \begin{array}{c} (\tilde{X}_1'\tilde{X}_1)^{-1}\tilde{X}_1'\tilde{y} \\ (\tilde{X}_2'\tilde{X}_2)^{-1}\tilde{X}_2'\tilde{y} \\ \end{array}% \right) \end{eqnarray*} $$ Thus, the residuals are $$ \tilde{e}=\tilde{y}-(\tilde{X}_1:\tilde{X}_2)b=\tilde{y}-(P_{\tilde{X}_1}\tilde{y}+P_{\tilde{X}_2}\tilde{y}), $$ so that, using idempotency of $P$ as well as $P_{\tilde{X}_1}P_{\tilde{X}_2}=0$, $$ \tilde{e}'\tilde{e}=\tilde{y}'\tilde{y}-\tilde{y}'P_{\tilde{X}_1}\tilde{y}-\tilde{y}'P_{\tilde{X}_2}\tilde{y}, $$ so that $$ R^2=\frac{\tilde{y}'\tilde{y}-\tilde{y}'\tilde{y}+\tilde{y}'P_{\tilde{X}_1}\tilde{y}+\tilde{y}'P_{\tilde{X}_2}}{\tilde{y}'\tilde{y}} $$

Numerical illustration:

n <- 5
x1 <- rnorm(n)
x2 <- rnorm(n)
x1.c <- resid(lm(x1~x2)) # to get a regressor uncorrelated to x2 
y <- rnorm(n)

Output:

> # centered case
> summary(lm(y~x1.c))$r.squared + summary(lm(y~x2))$r.squared 
[1] 0.2187793

> summary(lm(y~x1.c+x2))$r.squared
[1] 0.2187793

> # uncentered case
> summary(lm(y~x1.c-1))$r.squared + summary(lm(y~x2-1))$r.squared 
[1] 0.1250624

> summary(lm(y~x1.c+x2-1))$r.squared
[1] 0.1250624
$\endgroup$
1
  • 3
    $\begingroup$ If two regressors each explain more 50% of the variance, the $(X1,X2,Y)$ correlation matrix has two pairs of off-diagonal entries larger than $1/\sqrt{2}$. The remaining pair can't be zero or the matrix would not be positive semidefinite. $\endgroup$ Jun 9 '20 at 7:16
1
$\begingroup$

Usually you can't add these percentages to get the variance in $Z$ explained by $X$ and $Y$, because it might be the case that $X$ does not explain any variance in $Z$ beyond what is already explained by $Y$.

But in that case $X$ and $Y$ should be strongly correlated. So is the absence of a correlation between $X$ and $Y$ enough to allow us to add the percentages of variance explained or is there something that I'm missing?

You are allowed to add up the variance explained, if there are no other regressors.

If $X$ and $Y$ perpendicular to other regressors

If there are other regressors, but they are also perpendicular then you can still make a sort of similar summation of the variance explained.

Let's assume that the $X$ and $Y$ are perpendicular to the other regressors as well. Then we have

$$Z = a X + b Y + \text{other regressors} + \epsilon_{X+Y}$$

and since all these terms are perpendicular their variances add up.

$$\text{var}(Z) = \underbrace{\text{var}(a X) + \text{var}(b Y) + \text{var}(\text{other regressors})}_{\text{explained variance by model}} + \underbrace{\text{var}(\epsilon_{X+Y})}_{\text{error variance}}$$

When X, Y and the other regressors are perpendicular then their coefficients ($a$ and $b$) will not be different depending on whether you use them together or in single models. So the explained variances by the different models will be:

$$\begin{array}{rcl} \text{variance $X$ and $Y$ model:} & \text{var}(aX) + \text{var}(bY) +\text{var}(\text{other regressors})\\ \text{variance only $X$ model:} & \text{var}(aX) + \hphantom{\text{var}(bY)+} \text{var}(\text{other regressors})\\ \text{variance only $Y$ model:} & \hphantom{\text{var}(aX) +} \text{var}(bY) +\text{var}(\text{other regressors}) \end{array}$$

So if you have those other regressors then the variance explained by the models with single $X$ and $Y$ can not be attributed only to those $X$ and $Y$ and you can not add them together. You can only add the parts together that $X$ and $Y$ make independently from the other regressors.

$$R_{X+Y+\text{other}}^2 = R_{X+\text{other}}^2 + \underbrace{(R_{Y+\text{other}}^2 - R_{\text{other}}^2)}_{\text{the difference made by $Y$}} $$

If not $X$ and $Y$ perpendicular to other regressors

In this case the terms $\text{var(other regressors)}$, $\text{var($aX$)}$ and $\text{var($bY$)}$ will not remain the same in the different models. And there is no simple solution to add together the explained variance even if you correct for the other regressors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.