6
$\begingroup$

I understand that for simple linear regression, the sample correlation coefficient is the square root of the $R^2$. But that's just for a simple (i.e., single variable) regression $Y=\beta_0+\beta_1X+\varepsilon$.

How about multiple regression, e.g., $Y=\beta_0+\beta_1X_1 + \beta_2X_2+\varepsilon$? Is there any relationship between the correlations $corr(Y, X_1)$, $corr(Y, X_2)$ and the regression $R^2$?

$\endgroup$
  • $\begingroup$ What are you looking for in terms of "any relationship"? Someone liberally interpreting this question would point out, for example, that your recent related question provides just such a relationship, namely the joint $R^2$ is at least as large as the squared correlations between any of the individual predictors. $\endgroup$ – cardinal Jul 14 '12 at 18:58
  • $\begingroup$ possible duplicate of question on the possible range of $R^2$ $\endgroup$ – Macro Jul 14 '12 at 19:36
6
$\begingroup$

For two predictors, it is easy to write out the equation in algebraic form:

$R^2 = \frac{r^2_{x1,y} + r^2_{x2,y} - 2r_{x1,y}r_{x2,y}r_{x1,x2}}{1-r^2_{x1,x2}}$.

As pointed out by @gung, you also need to know the correlation between $x1$ and $x2$.

EDIT: Just a quick example (in R) to illustrate this equation:

set.seed(12873)

x1 <- rnorm(20)
x2 <- .1*x1 + rnorm(20)
y  <- .8*x1 + .2*x2 + rnorm(20)

summary(lm(y ~ x1 + x2))$r.square
(cor(x1,y)^2 + cor(x2,y)^2 - 2*cor(x1,y)*cor(x2,y)*cor(x1,x2))/(1-cor(x1,x2)^2)

gives the exact same answer of 0.2928677.

$\endgroup$
3
$\begingroup$

I'm sure there is, but I don't know it. Moreover it would be fairly complex, because you would also have to account for the correlations amongst the covariates. That is, you would need $corr(X_1,X_2)$ as well. The resulting formula would need to be expressed in matrix notation to make it possible to fit on one line and be able to account for any number of covariates, but that might also make it more opaque for most people unless they are very literate with matrix notation.

One simple thing I can say is that your regression model will yield a single vector of predicted values, $\hat y$. Under the assumption that your model has an intercept, the correlation between these values and the observed ones, $corr(\hat y,y)$, will always equal the square root of $R^2$, no matter how many covariates you have.

$\endgroup$
  • 1
    $\begingroup$ except when the correlation is negative ;) It is better to say that the squared correlation between the observations and the predicted values is the R squared $\endgroup$ – Stéphane Laurent Jul 14 '12 at 18:46
  • 2
    $\begingroup$ That's a good point, @StéphaneLaurent, I often slip & think only in terms of positive correlations. However, in this case, a regression model will only return betas such that the predicted values are positively correlated with the observations. Thanks for keeping me on my toes! $\endgroup$ – gung Jul 14 '12 at 18:49
  • 2
    $\begingroup$ @cardinal, you're right about that. I edited the answer to note that the intercept is required. $\endgroup$ – gung Jul 14 '12 at 18:54
3
$\begingroup$

If you compute the inverse of the correlation matrix of a set of variables, then take one minus the reciprocal of the diagonal elements ($1-\frac{1}{r_{ii}}$) then the results are the same as the $R^2$ value for the given term as response variable and all others as predictors.

> summary(lm( Sepal.Length ~ .-Species, data=iris ))$r.squared
    [1] 0.8586117
    > summary(lm( Sepal.Width ~ .-Species, data=iris ))$r.squared
[1] 0.5240071
> summary(lm( Petal.Length ~ .-Species, data=iris ))$r.squared
    [1] 0.9680118
    > summary(lm( Petal.Width ~ .-Species, data=iris ))$r.squared
[1] 0.9378503
> 1-1/diag(solve(cor(iris[,-5])))
Sepal.Length  Sepal.Width Petal.Length  Petal.Width 
   0.8586117    0.5240071    0.9680118    0.9378503 
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.