7
$\begingroup$

In simple linear regression, $R^2$ is equivalent to the squared correlation of a dependent and an independent variable. Is this also true for multiple linear regression?

For example, I measured trait openness to predict creativity in a simple linear regression. If I square the measured correlation between the two, I get the coefficient of determination.

Then I have measured the traits extraversion, openness and intellect to predict creativity in a multiple linear regression. Can I take those observed correlations, square and add them, and get the coefficent of determination for this kind of regression too?

$\endgroup$
5
  • 1
    $\begingroup$ I believe one version of this is answered (and extensively analyzed) at stats.stackexchange.com/questions/223818. But it depends on what you mean by "this"--what generalization do you have in mind? What statistics do you hope to compute $R^2$ from, exactly? $\endgroup$
    – whuber
    Nov 21, 2017 at 15:36
  • $\begingroup$ Thank you, from what I gathered there, it is not possible for regression coefficients, but I'm asking for correlations alone (not necessarily trying to fit a line). I edited my post to better describe my question. $\endgroup$
    – E_H
    Nov 21, 2017 at 17:20
  • $\begingroup$ When all the regressors are mutually orthogonal, such a formula is possible, but in general the result depends on the correlations among the regressors, too. $\endgroup$
    – whuber
    Nov 21, 2017 at 18:08
  • $\begingroup$ Please see footnote in stats.stackexchange.com/a/43224/3277. Sq. multiple correlations are the diagonal entries of image covariance matrix obtained from correlation matrix. $\endgroup$
    – ttnphns
    Nov 22, 2017 at 13:09
  • $\begingroup$ For alternative formulae of $R^2$, see stats.stackexchange.com/a/443899/119261. $\endgroup$ Jan 9, 2020 at 16:19

2 Answers 2

11
$\begingroup$

The coefficient-of-determination can be determined from the correlations: Consider a multiple linear regression with $m$ explanatory vectors and an intercept term. First we define the correlation values for all the variables in the problem $r_i = \mathbb{Corr}(\mathbf{y},\mathbf{x}_i)$ and $r_{i,j} = \mathbb{Corr}(\mathbf{x}_i,\mathbf{x}_j)$. Now define the goodness of fit vector and design correlation matrix respectively by:

$$\boldsymbol{r}_{\mathbf{y},\mathbf{x}} = \begin{bmatrix} r_1 \\ r_2 \\ \vdots \\ r_m \end{bmatrix} \quad \quad \quad \boldsymbol{r}_{\mathbf{x},\mathbf{x}} = \begin{bmatrix} r_{1,1} & r_{1,2} & \cdots & r_{1,m} \\ r_{2,1} & r_{2,2} & \cdots & r_{2,m} \\ \vdots & \vdots & \ddots & \vdots \\ r_{m,1} & r_{m,2} & \cdots & r_{m,m} \\ \end{bmatrix}.$$

The goodness-of-fit vector contains the correlations between the response vector and each of the explanatory vectors. The design correlation matrix contains the correlations between each pair of explanatory vectors. (Please note that these names are something I have made up, since neither matrix has a standard name that I am aware of. The first vector measures the goodness-of-fit of simple regressions on each of the individual explanatory vectors, which is why I use this name.) Now, with a bit of linear algebra it can be shown that the coefficient-of-determination for the multiple linear regression is given by the following quadratic form:

$$R^2 = \boldsymbol{r}_{\mathbf{y},\mathbf{x}}^\text{T} \boldsymbol{r}_{\mathbf{x},\mathbf{x}}^{-1} \boldsymbol{r}_{\mathbf{y},\mathbf{x}}.$$

This form for the coefficient-of-determination is not all that well-known to statistical practitioners, but it is a very useful result, and assists in framing the goodness-of-fit of the multiple linear regression in its most fundamental terms. The square-root of the coefficient of determination gives us the multiple correlation coefficient, which is a multivariate extension of the absolute correlation. In the special case where $m=1$ you get $R^2 = r_1^2$ so that the coefficient-of-determination is the square of the correlation between the response vector and the (single) explanatory variable.

As you can see, this form for the coefficient-of-determination for the multiple linear regression is framed fully in terms of correlations between the pairs of vectors going into the regression. This means that if you have a matrix of the pairwise correlations between all the vectors in the multiple regression (the response vector and each of the explanatory vectors) then you can directly determine the coefficient-of-determination without fitting the regression model. This result is more commonly presented in multivariate analysis (see e.g., Mardia, Kent and Bibby 1979, p. 168).


The coefficient-of-determination is not generally equal to the sum of individual coefficients: In the case where all the explanatory vectors are uncorrelated with each other you get $\boldsymbol{r}_{\mathbf{x},\mathbf{x}} = \boldsymbol{I}$ which means that the above quadratic form reduces to $R^2 = \sum r_i^2$. However, this is a special case that only arises in practice in cases where the explanatory variables are set by the researcher. The explanatory variables are not generally uncorrelated, and so the coefficient-of-determination is determined by the above quadratic form.

It is also useful to note that the coefficient-of-determination in a multiple linear regression can be above or below the sum of the individual coefficients-of-determination for corresponding simple linear regressions. Usually it is below this sum (since the total explanatory power is usually less than the sum of its parts) but sometimes it is above this sum.

$\endgroup$
5
  • $\begingroup$ Thanks for the nice explanation! Are correlations of interaction terms, for example x1 * x2 or x1^2 * x3, included in the vectors and corr. matrix, or do you only consider single terms like x1, x2 and x3? $\endgroup$
    – JE_Muc
    Mar 3, 2019 at 11:23
  • $\begingroup$ I posted this question as a follow up question. $\endgroup$
    – JE_Muc
    Mar 4, 2019 at 10:52
  • $\begingroup$ @Scotty1: You would include all terms in the design matrix, including the interaction terms. $\endgroup$
    – Ben
    Sep 6, 2020 at 22:33
  • $\begingroup$ Would be nice to see a proof of $R^2$ as a quadratic form. I don't see why that's the case. $\endgroup$
    – Kashif
    Jul 13, 2022 at 17:44
  • $\begingroup$ @Glassjawed: For derivation of this result, see e.g., O'Neill (2019) (pp. 3-9). $\endgroup$
    – Ben
    Jul 15, 2022 at 8:59
-1
$\begingroup$

If you have both the absolute errors from the regression and the dependent variable values, you can calculate $R^2$ as:

$$1 - ({\rm absolute\ error\ variance} / {\rm dependent\ variable\ variance})$$

Using the above, the number of independent variables or their correlations is not needed, which is quite handy for multiple linear regression.

$\endgroup$
2
  • 1
    $\begingroup$ -1 because this does not answer the question $\endgroup$ Aug 30, 2018 at 2:20
  • $\begingroup$ It is kind of you to review my answer from over six months ago, thank you for the review. $\endgroup$ Aug 30, 2018 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.