6
$\begingroup$

I am using IDL regression function to compute the multiple linear correlation coefficient...

x = [transpose(anom1),transpose(anom2),transpose(anom3)]
coef = regress(x,y, const=a0, correlation=corr, mcorrelation=mcorr, sigma=stderr)

mcorr is returned with values between 0.0 and 1.0. Clearly, the result of the following IDL code I dug out of the can can't be negative...

mcorrelation = SQRT(mult_covariance)

But, could you help me verify this? I would like a citation or to see 'the math' behind calculating mcorrelation.

$\endgroup$
7
$\begingroup$

The multiple correlation in standard linear regression cannot be negative, the maths are easy to show it, although it depends on what "multiple correlation" is taken to mean. The usual way you would calculate $R^{2}$ is:

$$R^2=\frac{SSR}{TSS}$$ where $$ SSR = \sum_{i} (\hat{Y_i}-\bar{Y})^2$$ and $$ TSS = \sum_{i} (Y_i-\bar{Y})^2$$

Since sums of squares can never be negative, neither can the $R^2$ value, as long as its calculated this way.

However, $R^2$ calculated this way can be greater than 1, if you use an estimator which does not have the observed residuals sum to zero. Or mathematically, $R^2$ will be necessarily bounded by 1 if

$$\sum_{i} (\hat{Y_i}-Y_i)=0$$ and $$\sum_{i} (\hat{Y_i}-Y_i)\hat{Y_i}=0$$

Or in words, the average of the residuals is equal to 0, and the fitted values are uncorrelated with the residuals over the whole data set.

This is because you can expand TSS as follows

$$ TSS = \sum_{i} (Y_i-\bar{Y})^2 = \sum_{i} ([Y_i-\hat{Y_i}]-[\bar{Y}-\hat{Y_i}])^2$$ $$=\sum_{i} (Y_i-\hat{Y_i})^2-2\sum_{i} [Y_i-\hat{Y_i}][\bar{Y}-\hat{Y_i}]+\sum_{i} (\bar{Y}-\hat{Y_i})^2$$ $$=\sum_{i} (Y_i-\hat{Y_i})^2-2\bar{Y}\sum_{i} [Y_i-\hat{Y_i}]+2\sum_{i} [Y_i-\hat{Y_i}]\hat{Y_i}+\sum_{i} (\bar{Y}-\hat{Y_i})^2$$ $$=\sum_{i} (Y_i-\hat{Y_i})^2+\sum_{i} (\bar{Y}-\hat{Y_i})^2$$ $$\implies TSS=SSR+\sum_{i} (Y_i-\hat{Y_i})^2 \geq SSR \geq 0$$ $$\implies 1 \geq \frac{SSR}{TSS}=R^2 \geq 0$$

The constraints listed are always satisfied by the usual OLS estimators (in fact they form part of the equations that define OLS estimation)

$R^2$ can go negative if it is calculated by $1-\frac{SSE}{TSS}$ Where $SSE=\sum_{i} (Y_i-\hat{Y_i})^2$ instead of the way I described.

As a (silly) example of $R^2>1$, you can put as the estimate $\hat{Y_i}=\bar{Y_i}+TSS$ So That $SSR=n(TSS)^2$ and $R^2=n(TSS)$ Which will exceed 1 for big enough n or TSS. To make $R^2$ go negative, set $\hat{Y_i}=Y_i+TSS$ so that $SSE=n(TSS)^2$ and $R^2=1-n(TSS)$ which will be less than 0 for big enough n and TSS

$\endgroup$
  • $\begingroup$ what do you mean by "in fact they form part of the equations that define OLS estimation"? You can deduct the $\hat{y}$ without those equations. In fact those two equations are implied after we calculate $\hat{y}$, they are not apriori. $\endgroup$ – Theta30 Jul 13 '11 at 0:50
  • $\begingroup$ The objective function is $U(\beta)=\sum_i (Y_i-\hat{Y_i}(\beta))^{2}$ We differentiate w.r.t each beta $\frac{\partial U}{\partial \beta_j}=-2\sum_i X_{ij}(Y_i-\hat{Y_i}(\beta))$. We set to zero, so I can multiply the jth equation by $\beta_j$ (still zero on RHS), and I can add together all of the $p+1$ equations (still zero on RHS), and we have $-2\sum_j\beta_j\sum_i X_{ij}(Y_i-\hat{Y_i}(\beta))=0$. interchange sums $-2\sum_i (Y_i-\hat{Y_i}(\beta))\sum_j\beta_j X_{ij}=0$, divide by $-2$ and we have what I said as the result, $\sum_i (Y_i-\hat{Y_i}(\beta))\hat{Y_i}(\beta)=0$. $\endgroup$ – probabilityislogic Jul 13 '11 at 1:15
  • $\begingroup$ ok, thanks, that's what I basically said. I thought you meant "they are given before doing the OLS estimation" $\endgroup$ – Theta30 Jul 13 '11 at 1:33
1
$\begingroup$

r can indeed be negative - if two variables are negatively related. r^2 can only be between 0 and 1, for the simple reason that it is the square of a real number.

For example, if we correlated income and time spent in jail throughout life, I would guess we would get a negative correlation (I haven't done this, I'm just guessing).

$\endgroup$
  • 5
    $\begingroup$ $R^2$ can also be negative. This happens when other methods are used to calculate the coefficients in linear regression model specification, and the usual $R^2$ formula is used. I think $R^2$ can also get negative if regression is without the intercept. $\endgroup$ – mpiktas Jan 12 '11 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy