What is the interpretation of the "traditional" $R^2$?

Question

Suppose the following data correspond to observed responses and their predictions obtained from some model.

observed  <- runif(100)
predicted <- 1 + observed + rnorm(100, sd = 0.1)  # suppose they are obtained from some model instead

# predicted vs actual plot
plot(observed, predicted)

We can observe a linear trend and therefore expect a high $R^2$. $R^2$ can be interpreted as

• the degree of linear association between "predicted" and "observed", or
• the proportion of the total variation of the y-variable (here, "predicted") explained by the x-variable (here, "observed") using a straight line.

Question

Below are two versions of $R^2$ reported by caret. How do you interpret the "traditional" version, which gives -11 for the above example?

---

R2_default <- caret::R2(pred = predicted, obs = observed)
R2_tradi   <- caret::R2(pred = predicted, obs = observed, form = "traditional")
R2_tradi2  <- 1 - sum((observed - predicted)^2) / sum((observed - mean(observed))^2)

Answer 1

The default caret approach is form = 'corr', in which the square of the correlation coefficient between predicted and observed is calculated. This prevents any possible negative $R^2$ values. 'form = traditional' is 1 - (the sum of squared differences between the true value and the prediction / sum of squared differences between the observed value and the mean of all values). This theoretically allows for negative values, if the model predictions are on average worse than just predicting the average value.

In your case, you have a very strong correlation in your plot, but the precise values on the X- and Y-axes are very different because you added 1 to all 'predicted' values in the simulation. This makes the predictions very bad, which can be easily seen if we replot this with better axis limits and add a 1:1 line indicating where predictions and observations would be identical:

This large deviation from the diagonal is a bias that makes the model predictions worse than just predicting the mean (based on mean squared error, or really any criterion in this case) even though the predictions are highly correlated with the observed values.

This is a weird situation but I have encountered models with negative $R^2$ with real data, though not because of such a simple bias/offset. So I personally prefer the 'traditional' calculation and believe it provides a more accurate picture of model performance. Visualisation always communicates a more complete picture than a summary statistic, though, so I would still recommend examining a plot of predicted vs. observed.

You might be interested in these related threads:

Is $R^2$ useful or dangerous?

What does negative R-squared mean?

Manually calculated $R^2$ doesn't match up with randomForest() $R^2$ for testing new data

Also note that the function you are using is deprecated, use postResample instead.

Answer 2

In ordinary least squares linear regression with just one feature and an intercept (so “simple” linear regression), there are a number of calculations that yield the same result.

1. $(cor(x,y))^2$

2. $(cor(\hat y, y))^2$

3. $1 -\dfrac{\sum_{i=1}^n \left( y_i-\hat y_i \right)^2 }{\sum_{i=1}^n \left( y_i-\bar y \right)^2 }$

Consequently, any of these three can justifiably be called $R^2$.

Even moving to regression with multiple features, option #1 is not viable, but the other two remain viable calculations for any poring of truth values and predicted values.

However, when you obtain the predictions by a method other than ordinary least squares, options #2 and #3 need not be equal.

Your software is giving you the ability to pick which of those two calculations you want to perform. Option #2 is called the “corr” method, while option #3 is called the “traditional” method.

For reasons I discuss here, it is the third calculation (equivalent to #4 in the link) that makes sense to me as a comparison of your modeling of the conditional expected value compared to a baseline model of the conditional expected value that naïvely guesses the pooled/marginal expected value $\bar y$ every time; pay particular attention to the simulation that shows how correlation-based $R^2$ can mislead you into thinking your predictions are good. If you are in a setting where #2 and #3 are equivalent, then feel free to do the easier correlation-based calculation, but, in my view, that is only because of the equivalence, not because of inherent value to the correlation-based calculations.

(I can see some value to the correlation-based calculations in the sense that, a high correlation-based calculation and low (even negative) value of #3 could suggest a simple mapping between your poor predictions and good predictions, though I would wonder why I didn’t get good predictions at the beginning.)