3
$\begingroup$

In simple linear regression, $R^2$ is equivalent to the squared correlation of a dependent and an independent variable. Is this also true for multiple linear regression?

For example, I measured trait openness to predict creativity in a simple linear regression. If I square the measured correlation between the two, I get the coefficient of determination.

Then I have measured the traits extraversion, openness and intellect to predict creativity in a multiple linear regression. Can I take those observed correlations, square and add them, and get the coefficent of determination for this kind of regression too?

$\endgroup$
  • 1
    $\begingroup$ I believe one version of this is answered (and extensively analyzed) at stats.stackexchange.com/questions/223818. But it depends on what you mean by "this"--what generalization do you have in mind? What statistics do you hope to compute $R^2$ from, exactly? $\endgroup$ – whuber Nov 21 '17 at 15:36
  • $\begingroup$ Thank you, from what I gathered there, it is not possible for regression coefficients, but I'm asking for correlations alone (not necessarily trying to fit a line). I edited my post to better describe my question. $\endgroup$ – E_H Nov 21 '17 at 17:20
  • $\begingroup$ When all the regressors are mutually orthogonal, such a formula is possible, but in general the result depends on the correlations among the regressors, too. $\endgroup$ – whuber Nov 21 '17 at 18:08
  • $\begingroup$ Please see footnote in stats.stackexchange.com/a/43224/3277. Sq. multiple correlations are the diagonal entries of image covariance matrix obtained from correlation matrix. $\endgroup$ – ttnphns Nov 22 '17 at 13:09
2
$\begingroup$

The coefficient-of-determination can be determined from the correlations: Consider a multiple linear regression with $m$ explanatory vectors and an intercept term. First we define the correlation values for all the variables in the problem $r_i = \mathbb{Corr}(\mathbf{y},\mathbf{x}_i)$ and $r_{i,j} = \mathbb{Corr}(\mathbf{x}_i,\mathbf{x}_j)$. Now define the goodness of fit vector and design correlation matrix respectively by:

$$\boldsymbol{r}_{\mathbf{y},\mathbf{x}} = \begin{bmatrix} r_1 \\ r_2 \\ \vdots \\ r_m \end{bmatrix} \quad \quad \quad \boldsymbol{r}_{\mathbf{x},\mathbf{x}} = \begin{bmatrix} r_{1,1} & r_{1,2} & \cdots & r_{1,m} \\ r_{2,1} & r_{2,2} & \cdots & r_{2,m} \\ \vdots & \vdots & \ddots & \vdots \\ r_{m,1} & r_{m,2} & \cdots & r_{m,m} \\ \end{bmatrix}.$$

The goodness-of-fit vector contains the correlations between the response vector and each of the explanatory vectors. The design correlation matrix contains the correlations between each pair of explanatory vectors. (Please note that these names are something I have made up, since neither matrix has a standard name that I am aware of. The first vector measures the goodness-of-fit of simple regressions on each of the individual explanatory vectors, which is why I use this name.) Now, with a bit of linear algebra it can be shown that the coefficient-of-determination for the multiple linear regression is given by the following quadratic form:

$$R^2 = \boldsymbol{r}_{\mathbf{y},\mathbf{x}}^\text{T} \boldsymbol{r}_{\mathbf{x},\mathbf{x}}^{-1} \boldsymbol{r}_{\mathbf{y},\mathbf{x}}.$$

This form for the coefficient-of-determination is not all that well-known to statistical practitioners, but it is a very useful result, and assists in framing the goodness-of-fit of the multiple linear regression in its most fundamental terms. The square-root of the coefficient of determination gives us the multiple correlation coefficient, which is a multivariate extension of the absolute correlation. In the special case where $m=1$ you get $R^2 = r_1^2$ so that the coefficient-of-determination is the square of the correlation between the response vector and the (single) explanatory variable.

As you can see, this form for the coefficient-of-determination for the multiple linear regression is framed fully in terms of correlations between the pairs of vectors going into the regression. This means that if you have a matrix of the pairwise correlations between all the vectors in the multiple regression (the response vector and each of the explanatory vectors) then you can directly determine the coefficient-of-determination without fitting the regression model. This result is more commonly presented in multivariate analysis (see e.g., Mardia, Kent and Bibby 1979, p. 168).


The coefficient-of-determination is not generally equal to the sum of individual coefficients: In the case where all the explanatory vectors are uncorrelated with each other you get $\boldsymbol{r}_{\mathbf{x},\mathbf{x}} = \boldsymbol{I}$ which means that the above quadratic form reduces to $R^2 = \sum r_i^2$. However, this is a special case that only arises in practice in cases where the explanatory variables are set by the researcher. The explanatory variables are not generally uncorrelated, and so the coefficient-of-determination is determined by the above quadratic form.

It is also useful to note that the coefficient-of-determination in a multiple linear regression can be above or below the sum of the individual coefficients-of-determination for corresponding simple linear regressions. Usually it is below this sum (since the total explanatory power is usually less than the sum of its parts) but sometimes it is above this sum.

$\endgroup$
  • $\begingroup$ Thanks for the nice explanation! Are correlations of interaction terms, for example x1 * x2 or x1^2 * x3, included in the vectors and corr. matrix, or do you only consider single terms like x1, x2 and x3? $\endgroup$ – Scotty1- Mar 3 at 11:23
  • $\begingroup$ I posted this question as a follow up question. $\endgroup$ – Scotty1- Mar 4 at 10:52
-1
$\begingroup$

If you have both the absolute errors from the regression and the dependent variable values, you can calculate $R^2$ as:

$$1 - ({\rm absolute\ error\ variance} / {\rm dependent\ variable\ variance})$$

Using the above, the number of independent variables or their correlations is not needed, which is quite handy for multiple linear regression.

$\endgroup$
  • 1
    $\begingroup$ -1 because this does not answer the question $\endgroup$ – Jake Westfall Aug 30 '18 at 2:20
  • $\begingroup$ It is kind of you to review my answer from over six months ago, thank you for the review. $\endgroup$ – James Phillips Aug 30 '18 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.