2
$\begingroup$

Suppose the following data correspond to observed responses and their predictions obtained from some model.

observed  <- runif(100)
predicted <- 1 + observed + rnorm(100, sd = 0.1)  # suppose they are obtained from some model instead

# predicted vs actual plot
plot(observed, predicted)

enter image description here

We can observe a linear trend and therefore expect a high $R^2$. $R^2$ can be interpreted as

  • the degree of linear association between "predicted" and "observed", or
  • the proportion of the total variation of the y-variable (here, "predicted") explained by the x-variable (here, "observed") using a straight line.

Question

Below are two versions of $R^2$ reported by caret. How do you interpret the "traditional" version, which gives -11 for the above example?


R2_default <- caret::R2(pred = predicted, obs = observed)
R2_tradi   <- caret::R2(pred = predicted, obs = observed, form = "traditional")
R2_tradi2  <- 1 - sum((observed - predicted)^2) / sum((observed - mean(observed))^2)

enter image description here

$\endgroup$
0

2 Answers 2

7
$\begingroup$

The default caret approach is form = 'corr', in which the square of the correlation coefficient between predicted and observed is calculated. This prevents any possible negative $R^2$ values. 'form = traditional' is 1 - (the sum of squared differences between the true value and the prediction / sum of squared differences between the observed value and the mean of all values). This theoretically allows for negative values, if the model predictions are on average worse than just predicting the average value.

In your case, you have a very strong correlation in your plot, but the precise values on the X- and Y-axes are very different because you added 1 to all 'predicted' values in the simulation. This makes the predictions very bad, which can be easily seen if we replot this with better axis limits and add a 1:1 line indicating where predictions and observations would be identical:

Replotting OP's figure of predicted vs. observed, but with identical scales on X- and Y- axes and a 1:1 line, to highlight the bias

This large deviation from the diagonal is a bias that makes the model predictions worse than just predicting the mean (based on mean squared error, or really any criterion in this case) even though the predictions are highly correlated with the observed values.

This is a weird situation but I have encountered models with negative $R^2$ with real data, though not because of such a simple bias/offset. So I personally prefer the 'traditional' calculation and believe it provides a more accurate picture of model performance. Visualisation always communicates a more complete picture than a summary statistic, though, so I would still recommend examining a plot of predicted vs. observed.

You might be interested in these related threads:

Is $R^2$ useful or dangerous?

What does negative R-squared mean?

Manually calculated $R^2$ doesn't match up with randomForest() $R^2$ for testing new data

Also note that the function you are using is deprecated, use postResample instead.

$\endgroup$
2
  • 1
    $\begingroup$ Why are they both called R2 while they measure two different things? When you see an R2 value reported in a paper, which R2 do you expect to see? Thank you for your response (+1)! $\endgroup$
    – user7064
    Aug 26, 2022 at 12:59
  • $\begingroup$ @user7064 There are multiple ways that R2 can be defined, and I agree that it's unfortunate that they don't have distinct names. To know for sure what is being calculated, you'll need to see if the paper explains their calculation or mentions the function they used, and then check the documentation. I would say it is very rare for papers to do this, though. In most cases these different formulae return very similar values, so it's not a major problem in practice. $\endgroup$
    – mkt
    Aug 26, 2022 at 13:06
2
$\begingroup$

In ordinary least squares linear regression with just one feature and an intercept (so “simple” linear regression), there are a number of calculations that yield the same result.

  1. $(cor(x,y))^2$

  2. $(cor(\hat y, y))^2$

  3. $1 -\dfrac{\sum_{i=1}^n \left( y_i-\hat y_i \right)^2 }{\sum_{i=1}^n \left( y_i-\bar y \right)^2 }$

Consequently, any of these three can justifiably be called $R^2$.

Even moving to regression with multiple features, option #1 is not viable, but the other two remain viable calculations for any poring of truth values and predicted values.

However, when you obtain the predictions by a method other than ordinary least squares, options #2 and #3 need not be equal.

Your software is giving you the ability to pick which of those two calculations you want to perform. Option #2 is called the “corr” method, while option #3 is called the “traditional” method.

For reasons I discuss here, it is the third calculation (equivalent to #4 in the link) that makes sense to me as a comparison of your modeling of the conditional expected value compared to a baseline model of the conditional expected value that naïvely guesses the pooled/marginal expected value $\bar y$ every time; pay particular attention to the simulation that shows how correlation-based $R^2$ can mislead you into thinking your predictions are good. If you are in a setting where #2 and #3 are equivalent, then feel free to do the easier correlation-based calculation, but, in my view, that is only because of the equivalence, not because of inherent value to the correlation-based calculations.

(I can see some value to the correlation-based calculations in the sense that, a high correlation-based calculation and low (even negative) value of #3 could suggest a simple mapping between your poor predictions and good predictions, though I would wonder why I didn’t get good predictions at the beginning.)

$\endgroup$
5
  • $\begingroup$ Thank you for your input (+1) $\endgroup$
    – user7064
    Aug 26, 2022 at 13:16
  • $\begingroup$ You need to square both correlations #1 and #2 to get $R^2$. $\endgroup$
    – Nick Cox
    Aug 26, 2022 at 14:05
  • 1
    $\begingroup$ I don't think the label "traditional" helps any one, as whose tradition are we talking about? I may be in a minority but long ago I was taught to think first of the correlation squared, and then of a calculating formula such as #3. So, there are. minimally. different traditions here. $\endgroup$
    – Nick Cox
    Aug 26, 2022 at 14:08
  • 1
    $\begingroup$ @NickCox Fixed, thanks. // I’m with you that the naming is odd, as I would think of squared correlation to be the “traditional” method of calculation. $\endgroup$
    – Dave
    Aug 26, 2022 at 14:17
  • $\begingroup$ Where does the notation come from: $R^2$ is the square of $R$, which is the multiple correlation. $\endgroup$
    – Nick Cox
    Aug 26, 2022 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.