When is R squared negative? [duplicate]

Question

My understanding is that $R^2$ cannot be negative as it is the square of R. However I ran a simple linear regression in SPSS with a single independent variable and a dependent variable. My SPSS output give me a negative value for $R^2$. If I was to calculate this by hand from R then $R^2$ would be positive. What has SPSS done to calculate this as negative?

R=-.395
R squared =-.156
B (un-standardized)=-1261.611

Code I've used:

DATASET ACTIVATE DataSet1. 
REGRESSION /MISSING LISTWISE /STATISTICS COEFF OUTS R ANOVA 
           /CRITERIA=PIN(.05) POUT(.10) /NOORIGIN 
           /DEPENDENT valueP /METHOD=ENTER ageP

I get a negative value. Can anyone explain what this means?

Answer 1

$R^2$ compares the fit of the chosen model with that of a horizontal straight line (the null hypothesis). If the chosen model fits worse than a horizontal line, then $R^2$ is negative. Note that $R^2$ is not always the square of anything, so it can have a negative value without violating any rules of math. $R^2$ is negative only when the chosen model does not follow the trend of the data, so fits worse than a horizontal line.

Example: fit data to a linear regression model constrained so that the $Y$ intercept must equal $1500$.

The model makes no sense at all given these data. It is clearly the wrong model, perhaps chosen by accident.

The fit of the model (a straight line constrained to go through the point (0,1500)) is worse than the fit of a horizontal line. Thus the sum-of-squares from the model $(SS_\text{res})$ is larger than the sum-of-squares from the horizontal line $(SS_\text{tot})$.

If $R^2$ is computed as $1 - \frac{SS_\text{res}}{SS_\text{tot}}$. (here, $SS_{res}$ = residual error.)
When $SS_\text{res}$ is greater than $SS_\text{tot}$, that equation could compute a negative value for $R^2$, if the value of the coeficient is greater than 1.

With linear regression with no constraints, $R^2$ must be positive (or zero) and equals the square of the correlation coefficient, $r$. A negative $R^2$ is only possible with linear regression when either the intercept or the slope are constrained so that the "best-fit" line (given the constraint) fits worse than a horizontal line. With nonlinear regression, the $R^2$ can be negative whenever the best-fit model (given the chosen equation, and its constraints, if any) fits the data worse than a horizontal line.

Bottom line: a negative $R^2$ is not a mathematical impossibility or the sign of a computer bug. It simply means that the chosen model (with its constraints) fits the data really poorly.

Answer 2

Have you forgotten to include an intercept in your regression? I'm not familiar with SPSS code, but on page 21 of Hayashi's Econometrics:

If the regressors do not include a constant but (as some regression software packages do) you nevertheless calculate $R^2$ by the formula

$R^2=1-\frac{\sum_{i=1}^{n}e_i^2}{\sum_{i=1}^{n}(y_i-\bar{y})^2}$

then the $R^2$ can be negative. This is because, without the benefit of an intercept, the regression could do worse than the sample mean in terms of tracking the dependent variable (i.e., the numerator could be greater than the denominator).

I'd check and make sure that SPSS is including an intercept in your regression.

Answer 3

This can happen if you have a time series that is N.i.i.d. and you construct an inappropriate ARIMA model of the form(0,1,0) which is a first difference random walk model with no drift then the variance (sum of squares - SSE ) of the residuals will be larger than the variance (sum of squares SSO) of the original series. Thus the equation 1-SSE/SSO will yield a negative number as SSE execeedS SSO . We have seen this when users simply fit an assumed model or use inadequate procedures to identify/form an appropriate ARIMA structure. The larger message IS that a model can distort (much like a pair of bad glasses ) your vision. Without having access to your data I would otherwise have a problem in explaining your faulty results. Have you brought this to the attention of IBM ?

The idea of an assumed model being counter-productive has been echoed by Harvey Motulsky. Great post Harvey !

Answer 4

Here's an explanation for those from the ML field: a negative R squared means that the model is predicting worse than the mean of the target values ($\bar{y}$). In other words, the mean squared error (MSE) of the model is higher than the MSE of a dummy estimator using the mean of the target values as the prediction ($Rˆ2 = 1-\frac{MSE(y,f)}{MSE(y,\bar{y})}$).

As a curiosity, it can happen a counter-intuitive situation there's a high correlation between $y$ (target value) and $f$ (prediction), but still a negative r-squared. I'll demonstrate this using python below:

from sklearn.metrics import r2_score
from scipy.stats import pearsonr
import numpy as np

# True values
y = np.array([10, 20, 30, 50, 90])

# Predictions
f = np.array([20, 40, 60, 75, 135])

# Calculate r-squared
r2 = r2_score(y, f)
print('R-Squared:', r2)
# Output: -0.012

corr = pearsonr(y, f)[0]
print('Pearson Correlation:', corr)
# Output: 0.992

# Here's a way to interpret: r2 is equivalent to the following

mean_squared_error = lambda y, f: np.mean((y-f)**2)
r2_eq = 1-mean_squared_error(y, f)/mean_squared_error(y, [y.mean()]*len(y))
print(
    'R-Squared (from equation):',
    r2_eq
)
# Output: -0.0124

# Hence it is negative as MSE given f is higher than MSE given y.mean() as f

Which prints:

R-Squared: -0.012499999999999956
Pearson Correlation: 0.9929674489269135
R-Squared (from equation): -0.012499999999999956