Relationship between $R^2$ and correlation coefficient

Question

Let's say I have two 1-dimensional arrays, $a_1$ and $a_2$. Each contains 100 data points. $a_1$ is the actual data, and $a_2$ is the model prediction. In this case, the $R^2$ value would be: $$ R^2 = 1 - \frac{SS_{res}}{SS_{tot}} \quad\quad\quad\quad\quad\ \ \quad\quad(1). $$ In the meantime, this would be equal to the square value of the correlation coefficient, $$ R^2 = (\text{Correlation Coefficient})^2 \quad (2). $$ Now if I swap the two: $a_2$ is the actual data, and $a_1$ is the model prediction. From equation $(2)$, because correlation coefficient does not care which comes first, the $R^2$ value would be the same. However, from equation $(1)$, $SS_{tot}=\sum_i(y_i - \bar y )^2$, the $R^2$ value will change, because the $SS_{tot}$ has changed if we switch $y$ from $a_1$ to $a_2$; in the meantime, $SS_{res}=\sum_i(y_i -f_i)^2$ does not change.

My question is: How can these contradict each other?

Edit:

1. I was wondering that, will the relationship in Eq. (2) still stand, if it is not a simple linear regression, i.e., the relationship between IV and DV is not linear (could be exponential / log)?

2. Will this relationship still stand, if the sum of the prediction errors does not equal zero?

Answer 1

This is true that $SS_{tot}$ will change ... but you forgot the fact that the regression sum of of squares will change as well. So let's consider the simple regression model and denote the Correlation Coefficient as $r_{xy}^2=\dfrac{S_{xy}^2}{S_{xx}S_{yy}}$, where I used the sub-index $xy$ to emphasize the fact that $x$ is the independent variable and $y$ is the dependent variable. Obviously, $r_{xy}^2$ is unchanged if you swap $x$ with $y$. We can easily show that $SSR_{xy}=S_{yy}(R_{xy}^2)$, where $SSR_{xy}$ is the regression sum of of squares and $S_{yy}$ is the total sum of squares where $x$ is independent and $y$ is dependent variable. Therefore: $$R_{xy}^2=\dfrac{SSR_{xy}}{S_{yy}}=\dfrac{S_{yy}-SSE_{xy}}{S_{yy}},$$ where $SSE_{xy}$ is the corresponding residual sum of of squares where $x$ is independent and $y$ is dependent variable. Note that in this case, we have $SSE_{xy}=b^2_{xy}S_{xx}$ with $b=\dfrac{S_{xy}}{S_{xx}}$ (See e.g. Eq. (34)-(41) here.) Therefore: $$R_{xy}^2=\dfrac{S_{yy}-\dfrac{S^2_{xy}}{S^2_{xx}}.S_{xx}}{S_{yy}}=\dfrac{S_{yy}S_{xx}-S^2_{xy}}{S_{xx}.S_{yy}}.$$ Clearly above equation is symmetric with respect to $x$ and $y$. In other words: $$R_{xy}^2=R_{yx}^2.$$ To summarize when you change $x$ with $y$ in the simple regression model, both numerator and denominator of $R_{xy}^2=\dfrac{SSR_{xy}}{S_{yy}}$ will change in a way that $R_{xy}^2=R_{yx}^2.$

Answer 2

One way of interpreting the coefficient of determination $R^{2}$ is to look at it as the Squared Pearson Correlation Coefficient between the observed values $y_{i}$ and the fitted values $\hat{y}_{i}$.

The complete proof of how to derive the coefficient of determination R2 from the Squared Pearson Correlation Coefficient between the observed values yi and the fitted values y^i can be found under the following link:

http://economictheoryblog.wordpress.com/2014/11/05/proof/

In my eyes it should be pretty easy to understand, just follow the single steps. I guess looking at it is essential to understand how the realtionship between the two key figures actually works.

Answer 3

In case of simple linear regression with only one predictor $R^2 = r^2 = Corr(x,y)^2$. But in multiple linear regression with more than one predictors the concept of correlation between the predictors and the response does not extend automatically. The formula gets:

$$R^2 = Corr(y_{estimated},y_{observed})^2$$

The square of the correlation between the response and the fitted linear model.

Answer 4

@Stat has provided a detailed answer. In my short answer I'll show briefly in somewhat different way what is the similarity and difference between $r$ and $r^2$.

$r$ is the standardized regression coefficient beta of $Y$ by $X$ or of $X$ by $Y$ and as such, it is a measure of the (mutual) effect size. Which is most clearly seen when the variables are dichotomous. Then $r$, for example, $.30$ means that 30% of cases will change its value to opposite in one variable when the other variable changes its value to the opposite.

$r^2$, on the other hand, is the expression of the proportion of co-variability in the total variability: $r^2 = (\frac {cov}{\sigma_x \sigma_y})^2 = \frac {|cov|} {\sigma_x^2} \frac {|cov|} {\sigma_y^2}$. Note that this is a product of two proportions, or, more precise to say, two ratios (a ratio can be >1). If loosely imply any proportion or ratio to be a quasi-probability or propensity, then $r^2$ expresses "joint probability (propensity)". Another and as valid expression for the joint product of two proportions (or ratios) would be their geometric mean, $\sqrt{prop*prop}$, which is very $r$.

(The two ratios are multiplicative, not additive, to stress the idea that they collaborate and cannot compensate for each other, in their teamwork. They have to be multiplicative because the magnitude of $cov$ is dependent on both magnitudes $\sigma_x^2$ and $\sigma_y^2$ and, conformably, $cov$ has to be divided two times in once - in order to convert itself to a proper "proportion of the shared variance". But $cov$, the "cross-variance", shares the same measurement units with both $\sigma_x^2$ and $\sigma_y^2$, the "self-variances", and not with $\sigma_x \sigma_y$, the "hybrid variance"; that is why $r^2$, not $r$, is more adequate as the "proportion of shared variance".)

So, you see that meaning of $r$ and $r^2$ as a measure of the quantity of the association is different (both meanings valid), but still these coefficients in no way contradict each other. And both are the same whether you predict $Y\text~X$ or $X\text~Y$.

Answer 5

I think you might be mistaken. If $R^2=r^2$, I assume you have a bivariate model: one DV, one IV. I don't think $R^2$ will change if you swap these, nor if you replace the IV with the predictions of the DV that are based on the IV. Here's code for a demonstration in R:

x=rnorm(1000); y=rnorm(1000)              # store random data
summary(lm(y~x))                          # fit a linear regression model (a)
summary(lm(x~y))                          # swap variables and fit the opposite model (b)
z=lm(y~x)$fitted.values; summary(lm(y~z)) # substitute predictions for IV in model (a)

If you aren't working with a bivariate model, your choice of DV will affect $R^2$...unless your variables are all identically correlated, I suppose, but this isn't much of an exception. If all the variables have identical strengths of correlation and also share the same portions of the DV's variance (e.g. [or maybe "i.e."], if some of the variables are completely identical), you could just reduce this to a bivariate model without losing any information. Whether you do or don't, $R^2$ still wouldn't change.

In all other cases I can think of with more than two variables, $R^2\ne r^2$ where $R^2$ is the coefficient of determination and $r$ is a bivariate correlation coefficient of any kind (not necessarily Pearson's; e.g., possibly also a Spearman's $\rho$).

Answer 6

If the prediction is not a projection onto the space spanned by the independent variables, the first definition is wrong. It can even be negative. The second definition is the same as the first if the prediction is a linear regression. Otherwise they are not the same. That said the second (correlation squared) definition always has meaning and ranges between zero and one.