60
$\begingroup$

Let's say I have two 1-dimensional arrays, $a_1$ and $a_2$. Each contains 100 data points. $a_1$ is the actual data, and $a_2$ is the model prediction. In this case, the $R^2$ value would be: $$ R^2 = 1 - \frac{SS_{res}}{SS_{tot}} \quad\quad\quad\quad\quad\ \ \quad\quad(1). $$ In the meantime, this would be equal to the square value of the correlation coefficient, $$ R^2 = (\text{Correlation Coefficient})^2 \quad (2). $$ Now if I swap the two: $a_2$ is the actual data, and $a_1$ is the model prediction. From equation $(2)$, because correlation coefficient does not care which comes first, the $R^2$ value would be the same. However, from equation $(1)$, $SS_{tot}=\sum_i(y_i - \bar y )^2$, the $R^2$ value will change, because the $SS_{tot}$ has changed if we switch $y$ from $a_1$ to $a_2$; in the meantime, $SS_{res}=\sum_i(y_i -f_i)^2$ does not change.

My question is: How can these contradict each other?

Edit:

  1. I was wondering that, will the relationship in Eq. (2) still stand, if it is not a simple linear regression, i.e., the relationship between IV and DV is not linear (could be exponential / log)?

  2. Will this relationship still stand, if the sum of the prediction errors does not equal zero?

$\endgroup$
1
  • $\begingroup$ I found this presentation very helpful and non technical: google.com/… $\endgroup$
    – ihadanny
    Sep 22, 2015 at 22:06

6 Answers 6

27
$\begingroup$

This is true that $SS_{tot}$ will change ... but you forgot the fact that the regression sum of of squares will change as well. So let's consider the simple regression model and denote the Correlation Coefficient as $r_{xy}^2=\dfrac{S_{xy}^2}{S_{xx}S_{yy}}$, where I used the sub-index $xy$ to emphasize the fact that $x$ is the independent variable and $y$ is the dependent variable. Obviously, $r_{xy}^2$ is unchanged if you swap $x$ with $y$. We can easily show that $SSR_{xy}=S_{yy}(R_{xy}^2)$, where $SSR_{xy}$ is the regression sum of of squares and $S_{yy}$ is the total sum of squares where $x$ is independent and $y$ is dependent variable. Therefore: $$R_{xy}^2=\dfrac{SSR_{xy}}{S_{yy}}=\dfrac{S_{yy}-SSE_{xy}}{S_{yy}},$$ where $SSE_{xy}$ is the corresponding residual sum of of squares where $x$ is independent and $y$ is dependent variable. Note that in this case, we have $SSE_{xy}=b^2_{xy}S_{xx}$ with $b=\dfrac{S_{xy}}{S_{xx}}$ (See e.g. Eq. (34)-(41) here.) Therefore: $$R_{xy}^2=\dfrac{S_{yy}-\dfrac{S^2_{xy}}{S^2_{xx}}.S_{xx}}{S_{yy}}=\dfrac{S_{yy}S_{xx}-S^2_{xy}}{S_{xx}.S_{yy}}.$$ Clearly above equation is symmetric with respect to $x$ and $y$. In other words: $$R_{xy}^2=R_{yx}^2.$$ To summarize when you change $x$ with $y$ in the simple regression model, both numerator and denominator of $R_{xy}^2=\dfrac{SSR_{xy}}{S_{yy}}$ will change in a way that $R_{xy}^2=R_{yx}^2.$

$\endgroup$
3
  • 1
    $\begingroup$ Thank you so much! I noticed that this might be where I was wrong: $R^2 = r^2$ only stands if 1) the model prediction is a straight line and 2) the mean of the model prediction equals the mean of the sample points. If the relationship between the DV and IV is not a straight line, or the sum of the prediction errors is non-zero, the relationship will not stand. Could you please let me know whether this is correct? $\endgroup$
    – Shawn Wang
    Jan 26, 2014 at 14:40
  • 2
    $\begingroup$ I thought about this because you have been using $R^2=SS_{reg}/SS_{tot}$, while I was using the equation I posted in the OP. These two equations are equivalent to each other only when the sum of the prediction errors is zero. Hence, in my OP, $SS_{res}=\sum_i(f_i-\bar y)^2$ does not change while $SS_{tot}$ changed, and hence the $R^2$ is changed. $\endgroup$
    – Shawn Wang
    Jan 26, 2014 at 14:49
  • $\begingroup$ Do you happen to have a reference for how to work this out for the general case of p-variate Gaussians? $\endgroup$
    – jmb
    May 2, 2018 at 10:38
36
$\begingroup$

One way of interpreting the coefficient of determination $R^{2}$ is to look at it as the Squared Pearson Correlation Coefficient between the observed values $y_{i}$ and the fitted values $\hat{y}_{i}$.

The complete proof of how to derive the coefficient of determination R2 from the Squared Pearson Correlation Coefficient between the observed values yi and the fitted values y^i can be found under the following link:

http://economictheoryblog.wordpress.com/2014/11/05/proof/

In my eyes it should be pretty easy to understand, just follow the single steps. I guess looking at it is essential to understand how the realtionship between the two key figures actually works.

$\endgroup$
13
$\begingroup$

In case of simple linear regression with only one predictor $R^2 = r^2 = Corr(x,y)^2$. But in multiple linear regression with more than one predictors the concept of correlation between the predictors and the response does not extend automatically. The formula gets:

$$R^2 = Corr(y_{estimated},y_{observed})^2$$

The square of the correlation between the response and the fitted linear model.

$\endgroup$
1
  • $\begingroup$ R^=r^2 only in the null model. If there is a regresion line R^2 measure cor(\hat{y},y) $\endgroup$
    – Freeman
    May 5, 2020 at 8:41
5
$\begingroup$

@Stat has provided a detailed answer. In my short answer I'll show briefly in somewhat different way what is the similarity and difference between $r$ and $r^2$.

$r$ is the standardized regression coefficient beta of $Y$ by $X$ or of $X$ by $Y$ and as such, it is a measure of the (mutual) effect size. Which is most clearly seen when the variables are dichotomous. Then $r$, for example, $.30$ means that 30% of cases will change its value to opposite in one variable when the other variable changes its value to the opposite.

$r^2$, on the other hand, is the expression of the proportion of co-variability in the total variability: $r^2 = (\frac {cov}{\sigma_x \sigma_y})^2 = \frac {|cov|} {\sigma_x^2} \frac {|cov|} {\sigma_y^2}$. Note that this is a product of two proportions, or, more precise to say, two ratios (a ratio can be >1). If loosely imply any proportion or ratio to be a quasi-probability or propensity, then $r^2$ expresses "joint probability (propensity)". Another and as valid expression for the joint product of two proportions (or ratios) would be their geometric mean, $\sqrt{prop*prop}$, which is very $r$.

(The two ratios are multiplicative, not additive, to stress the idea that they collaborate and cannot compensate for each other, in their teamwork. They have to be multiplicative because the magnitude of $cov$ is dependent on both magnitudes $\sigma_x^2$ and $\sigma_y^2$ and, conformably, $cov$ has to be divided two times in once - in order to convert itself to a proper "proportion of the shared variance". But $cov$, the "cross-variance", shares the same measurement units with both $\sigma_x^2$ and $\sigma_y^2$, the "self-variances", and not with $\sigma_x \sigma_y$, the "hybrid variance"; that is why $r^2$, not $r$, is more adequate as the "proportion of shared variance".)

So, you see that meaning of $r$ and $r^2$ as a measure of the quantity of the association is different (both meanings valid), but still these coefficients in no way contradict each other. And both are the same whether you predict $Y\text~X$ or $X\text~Y$.

$\endgroup$
4
  • $\begingroup$ Thank you so much! I am starting to wonder whether I am using the wrong definition, that two definitions of $R^2$ co-exist and they are not equivalent to each other. Could you please help me in the question that - if I am thinking about more generalized cases where the model is not a simple linear regression (could be exponential) - is my equation in the OP still correct for calculating $R^2$? Is this a different quantity, also called $R^2$, but different from the "coefficient of determination"? $\endgroup$
    – Shawn Wang
    Jan 26, 2014 at 14:54
  • $\begingroup$ Coefficient of determination or R-square is a wider concept than r^2 which is only about simple linear regression. Please read wikipedia en.wikipedia.org/wiki/Coefficient_of_determination. $\endgroup$
    – ttnphns
    Jan 26, 2014 at 19:11
  • $\begingroup$ Thanks again! That I do understand. My question is: for more complex regressions, can I still square the r value to get the coefficient of determination? $\endgroup$
    – Shawn Wang
    Jan 26, 2014 at 19:39
  • 1
    $\begingroup$ For a "complex regression", you get R-square, but you don't get r. $\endgroup$
    – ttnphns
    Jan 27, 2014 at 1:48
1
$\begingroup$

I think you might be mistaken. If $R^2=r^2$, I assume you have a bivariate model: one DV, one IV. I don't think $R^2$ will change if you swap these, nor if you replace the IV with the predictions of the DV that are based on the IV. Here's code for a demonstration in R:

x=rnorm(1000); y=rnorm(1000)              # store random data
summary(lm(y~x))                          # fit a linear regression model (a)
summary(lm(x~y))                          # swap variables and fit the opposite model (b)
z=lm(y~x)$fitted.values; summary(lm(y~z)) # substitute predictions for IV in model (a)

If you aren't working with a bivariate model, your choice of DV will affect $R^2$...unless your variables are all identically correlated, I suppose, but this isn't much of an exception. If all the variables have identical strengths of correlation and also share the same portions of the DV's variance (e.g. [or maybe "i.e."], if some of the variables are completely identical), you could just reduce this to a bivariate model without losing any information. Whether you do or don't, $R^2$ still wouldn't change.

In all other cases I can think of with more than two variables, $R^2\ne r^2$ where $R^2$ is the coefficient of determination and $r$ is a bivariate correlation coefficient of any kind (not necessarily Pearson's; e.g., possibly also a Spearman's $\rho$).

$\endgroup$
1
  • 1
    $\begingroup$ I recently did Theil linear regression then calculated $R^2=–0.1468$ and $SSR>SST$. I have seen Excel produce $-R^2$-values as well, and at first I laughed at it, then slowly came understanding and it ceased to be funny. So is the general definition of $R^2$ correct? What gives. $\endgroup$
    – Carl
    Aug 27, 2016 at 19:44
0
$\begingroup$

If the prediction is not a projection onto the space spanned by the independent variables, the first definition is wrong. It can even be negative. The second definition is the same as the first if the prediction is a linear regression. Otherwise they are not the same. That said the second (correlation squared) definition always has meaning and ranges between zero and one.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.