6
$\begingroup$

I am using IDL regression function to compute the multiple linear correlation coefficient...

x = [transpose(anom1),transpose(anom2),transpose(anom3)]
coef = regress(x,y, const=a0, correlation=corr, mcorrelation=mcorr, sigma=stderr)

mcorr is returned with values between 0.0 and 1.0. Clearly, the result of the following IDL code I dug out of the can can't be negative...

mcorrelation = SQRT(mult_covariance)

But, could you help me verify this? I would like a citation or to see 'the math' behind calculating mcorrelation.

$\endgroup$
2

2 Answers 2

10
$\begingroup$

The multiple correlation in standard linear regression cannot be negative, the maths are easy to show it, although it depends on what "multiple correlation" is taken to mean. The usual way you would calculate $R^{2}$ is:

$$R^2=\frac{SSR}{TSS}$$ where $$ SSR = \sum_{i} (\hat{Y_i}-\bar{Y})^2$$ and $$ TSS = \sum_{i} (Y_i-\bar{Y})^2$$

Since sums of squares can never be negative, neither can the $R^2$ value, as long as its calculated this way.

However, $R^2$ calculated this way can be greater than 1, if you use an estimator which does not have the observed residuals sum to zero. Or mathematically, $R^2$ will be necessarily bounded by 1 if

$$\sum_{i} (\hat{Y_i}-Y_i)=0$$ and $$\sum_{i} (\hat{Y_i}-Y_i)\hat{Y_i}=0$$

Or in words, the average of the residuals is equal to 0, and the fitted values are uncorrelated with the residuals over the whole data set.

This is because you can expand TSS as follows

$$ TSS = \sum_{i} (Y_i-\bar{Y})^2 = \sum_{i} ([Y_i-\hat{Y_i}]-[\bar{Y}-\hat{Y_i}])^2$$ $$=\sum_{i} (Y_i-\hat{Y_i})^2-2\sum_{i} [Y_i-\hat{Y_i}][\bar{Y}-\hat{Y_i}]+\sum_{i} (\bar{Y}-\hat{Y_i})^2$$ $$=\sum_{i} (Y_i-\hat{Y_i})^2-2\bar{Y}\sum_{i} [Y_i-\hat{Y_i}]+2\sum_{i} [Y_i-\hat{Y_i}]\hat{Y_i}+\sum_{i} (\bar{Y}-\hat{Y_i})^2$$ $$=\sum_{i} (Y_i-\hat{Y_i})^2+\sum_{i} (\bar{Y}-\hat{Y_i})^2$$ $$\implies TSS=SSR+\sum_{i} (Y_i-\hat{Y_i})^2 \geq SSR \geq 0$$ $$\implies 1 \geq \frac{SSR}{TSS}=R^2 \geq 0$$

The constraints listed are always satisfied by the usual OLS estimators (in fact they form part of the equations that define OLS estimation)

$R^2$ can go negative if it is calculated by $1-\frac{SSE}{TSS}$ Where $SSE=\sum_{i} (Y_i-\hat{Y_i})^2$ instead of the way I described.

As a (silly) example of $R^2>1$, you can put as the estimate $\hat{Y_i}=\bar{Y_i}+TSS$ So That $SSR=n(TSS)^2$ and $R^2=n(TSS)$ Which will exceed 1 for big enough n or TSS. To make $R^2$ go negative, set $\hat{Y_i}=Y_i+TSS$ so that $SSE=n(TSS)^2$ and $R^2=1-n(TSS)$ which will be less than 0 for big enough n and TSS

$\endgroup$
3
  • $\begingroup$ what do you mean by "in fact they form part of the equations that define OLS estimation"? You can deduct the $\hat{y}$ without those equations. In fact those two equations are implied after we calculate $\hat{y}$, they are not apriori. $\endgroup$ Commented Jul 13, 2011 at 0:50
  • 1
    $\begingroup$ The objective function is $U(\beta)=\sum_i (Y_i-\hat{Y_i}(\beta))^{2}$ We differentiate w.r.t each beta $\frac{\partial U}{\partial \beta_j}=-2\sum_i X_{ij}(Y_i-\hat{Y_i}(\beta))$. We set to zero, so I can multiply the jth equation by $\beta_j$ (still zero on RHS), and I can add together all of the $p+1$ equations (still zero on RHS), and we have $-2\sum_j\beta_j\sum_i X_{ij}(Y_i-\hat{Y_i}(\beta))=0$. interchange sums $-2\sum_i (Y_i-\hat{Y_i}(\beta))\sum_j\beta_j X_{ij}=0$, divide by $-2$ and we have what I said as the result, $\sum_i (Y_i-\hat{Y_i}(\beta))\hat{Y_i}(\beta)=0$. $\endgroup$ Commented Jul 13, 2011 at 1:15
  • $\begingroup$ ok, thanks, that's what I basically said. I thought you meant "they are given before doing the OLS estimation" $\endgroup$ Commented Jul 13, 2011 at 1:33
1
$\begingroup$

$R$ can indeed be negative - if two variables are negatively related. $R^2$ can only be between 0 and 1, for the simple reason that it is the square of a real number.

For example, if we correlated income and time spent in jail throughout life, I would guess we would get a negative correlation (I haven't done this, I'm just guessing).

$\endgroup$
3
  • 8
    $\begingroup$ $R^2$ can also be negative. This happens when other methods are used to calculate the coefficients in linear regression model specification, and the usual $R^2$ formula is used. I think $R^2$ can also get negative if regression is without the intercept. $\endgroup$
    – mpiktas
    Commented Jan 12, 2011 at 20:23
  • $\begingroup$ R2 can be negative if the model is performing worse than the Null model (intercept only model). $\endgroup$ Commented May 22, 2021 at 5:10
  • $\begingroup$ This is wrong. correlation can be negative or positive, if there is a correlation R^2 will be highly positive. $\endgroup$
    – PagMax
    Commented May 26, 2022 at 11:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.