8
$\begingroup$

I understand that for simple linear regression, the sample correlation coefficient is the square root of the $R^2$. But that's just for a simple (i.e., single variable) regression $Y=\beta_0+\beta_1X+\varepsilon$.

How about multiple regression, e.g., $Y=\beta_0+\beta_1X_1 + \beta_2X_2+\varepsilon$? Is there any relationship between the correlations $corr(Y, X_1)$, $corr(Y, X_2)$ and the regression $R^2$?

Improve this question
$\endgroup$
3
  • $\begingroup$ What are you looking for in terms of "any relationship"? Someone liberally interpreting this question would point out, for example, that your recent related question provides just such a relationship, namely the joint $R^2$ is at least as large as the squared correlations between any of the individual predictors. $\endgroup$
    – cardinal
    Jul 14, 2012 at 18:58
  • $\begingroup$ possible duplicate of question on the possible range of $R^2$ $\endgroup$
    – Macro
    Jul 14, 2012 at 19:36
  • $\begingroup$ stats.stackexchange.com/questions/351200/… $\endgroup$
    – StubbornAtom
    Jan 9, 2020 at 15:37

3 Answers 3

Reset to default
6
$\begingroup$

For two predictors, it is easy to write out the equation in algebraic form:

$$ \Large R^2 = \frac{r^2_{x1,y} + r^2_{x2,y} - 2r_{x1,y}r_{x2,y}r_{x1,x2}}{1-r^2_{x1,x2}} $$

As pointed out by @gung, you also need to know the correlation between $x1$ and $x2$.

EDIT: Just a quick example (in R) to illustrate this equation:

set.seed(12873)

x1 <- rnorm(20)
x2 <- .1*x1 + rnorm(20)
y  <- .8*x1 + .2*x2 + rnorm(20)

summary(lm(y ~ x1 + x2))$r.square
(cor(x1,y)^2 + cor(x2,y)^2 - 2*cor(x1,y)*cor(x2,y)*cor(x1,x2))/(1-cor(x1,x2)^2)

gives the exact same answer of 0.2928677.

Improve this answer
$\endgroup$
3
$\begingroup$

I'm sure there is, but I don't know it. Moreover it would be fairly complex, because you would also have to account for the correlations amongst the covariates. That is, you would need $corr(X_1,X_2)$ as well. The resulting formula would need to be expressed in matrix notation to make it possible to fit on one line and be able to account for any number of covariates, but that might also make it more opaque for most people unless they are very literate with matrix notation.

One simple thing I can say is that your regression model will yield a single vector of predicted values, $\hat y$. Under the assumption that your model has an intercept, the correlation between these values and the observed ones, $corr(\hat y,y)$, will always equal the square root of $R^2$, no matter how many covariates you have.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ except when the correlation is negative ;) It is better to say that the squared correlation between the observations and the predicted values is the R squared $\endgroup$
    – Stéphane Laurent
    Jul 14, 2012 at 18:46
  • 2
    $\begingroup$ That's a good point, @StéphaneLaurent, I often slip & think only in terms of positive correlations. However, in this case, a regression model will only return betas such that the predicted values are positively correlated with the observations. Thanks for keeping me on my toes! $\endgroup$
    – gung - Reinstate Monica
    Jul 14, 2012 at 18:49
  • 2
    $\begingroup$ @cardinal, you're right about that. I edited the answer to note that the intercept is required. $\endgroup$
    – gung - Reinstate Monica
    Jul 14, 2012 at 18:54
3
$\begingroup$

If you compute the inverse of the correlation matrix of a set of variables, then take one minus the reciprocal of the diagonal elements ($1-\frac{1}{r_{ii}}$) then the results are the same as the $R^2$ value for the given term as response variable and all others as predictors.

> summary(lm( Sepal.Length ~ .-Species, data=iris ))$r.squared
    [1] 0.8586117
    > summary(lm( Sepal.Width ~ .-Species, data=iris ))$r.squared
[1] 0.5240071
> summary(lm( Petal.Length ~ .-Species, data=iris ))$r.squared
    [1] 0.9680118
    > summary(lm( Petal.Width ~ .-Species, data=iris ))$r.squared
[1] 0.9378503
> 1-1/diag(solve(cor(iris[,-5])))
Sepal.Length  Sepal.Width Petal.Length  Petal.Width 
   0.8586117    0.5240071    0.9680118    0.9378503
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.