I've come across two ways that people calculate R-squared on a test set:

  1. Calculate the square of the correlation between predictions and actual values (in practice, I've seen people do this in R by regressing the test set Y on their test set predictions, and report the R-squared from R's lm summary)
  2. Use the sum of squares formula (the formula used by https://scikit-learn.org/stable/modules/model_evaluation.html#r2-score)

When applied to the training set of a linear regression, those two approaches are identical (see Relationship between $R^2$ and correlation coefficient for example), so there is no problem using one formula or the other.

However, on a test set, where the average prediction error might be nonzero (and you could even see extreme cases where predictions are negatively correlated with the ground truth, in addition to the average prediction error being nonzero), approaches 1 and 2 can produce different results. In extreme cases, approach 1 could yield a positive R-squared while approach 2 yields a negative R-squared (see What does negative R-squared mean? for examples). Is there a generally accepted way to refer to these two quantities, i.e. to name and disambiguate them, so that it's clear what people mean when they refer to a model's "test set R-squared"?

Related question: How to calculate out of sample R squared? is similar to my question but doesn't exactly answer how to disambiguate the two R-squared formulas.

In case the motivation isn't clear, here is an R simulation illustrating the difference between the two approaches using fake data:

## When computing R-squared on a test set, the result can differ meaningfully based on which formula we use
## This happens because average prediction error on a test set is often nonzero, especially in real-world problems
## Which formula do we want to use?  Should we report both versions of test set R-squared?
## Or, to keep things simple, should we report test set RMSEs and MAEs instead of R-squared?


n_obs <- 500

## This dataframe is a simulated _test_ set
## Imagine that the predictions come from a model trained on some other dataset (which, for simplicity, is not simulated here)
df <- data.frame(prediction=runif(n_obs, min=-5, max=5))

## Note that the addition of the constant in this line means that our
## average prediction error is _not_ zero
## This will often be the case on real-world test sets
df$actual = 1.25 + df$prediction + rnorm(n_obs, sd=2.5)

r2_general <- function(preds, actual) {
  ## Same formula as https://scikit-learn.org/stable/modules/model_evaluation.html#r2-score
  return(1 - sum((actual - preds) ^ 2) / sum((actual - mean(actual))^2))

## This is our average prediction error on the test set
## It will be close to the constant in the df$actual = ... equation
mean(df$actual - df$prediction)

## The model's predictions are positively correlated with the actual values,
## but, importantly, the average prediction error is nonzero
plot(df$prediction, df$actual)

cor(df$prediction, df$actual)

cor(df$prediction, df$actual)^2

regression <- lm(actual ~ prediction, data=df)

## The R-squared value in this regression output looks pretty good (R squared around 0.55),
## but that hides the fact that the average prediction error is nonzero
## Note that this R-squared is identical to cor(df$prediction, df$actual)^2

## This is lower than the R-squared from the regression output, because the average prediction error is nonzero
## This calculation gives 0.44 instead of 0.55 -- the test set R-squared depends on which formula we use
r2_general(preds=df$prediction, actual=df$actual)

df$prediction_from_regression <- predict(regression, newdata=df)

## Note that this agrees exactly with lm's R squared
## Fitting a regression on our test set creates a _new_ set of predictions,
## called prediction_from_regression, which differ from our original predictions
## Importantly, the average error of prediction_from_regression is zero
r2_general(preds=df$prediction_from_regression, actual=df$actual)
cor(df$prediction_from_regression, df$actual)^2

mean(df$actual - df$prediction)  # Not zero
mean(df$actual - df$prediction_from_regression)  # Zero
  • 1
    $\begingroup$ As far as I'm concerned, the justification for the term $R^2$ is within the standard linear regression model with intercept, on the data used for fitting, where it cannot be negative. Whenever people use the term outside that context, they should not assume that it is clear what it means, and define their use explicitly (or even better give it another name). I know some people just use it without explicit explanation, but it may be their fault, not the reader's, if they are not understood. $\endgroup$ Mar 30, 2022 at 21:17
  • $\begingroup$ Another related question+answer that could be helpful to someone reading this post: stats.stackexchange.com/a/551916/9330 $\endgroup$
    – Adrian
    Mar 30, 2022 at 21:35
  • $\begingroup$ @Adrian Could you please explain how my post about $R^2$ in the nonlinear case comes into play here? $\endgroup$
    – Dave
    Mar 30, 2022 at 21:52
  • $\begingroup$ @Dave yes, great question! The connection I see is that the "Other" term that you emphasize in your answer can be nonzero even in the case of a linear regression when we are evaluating that regression on the test set. In other words, we can be certain that the "Other" term in your answer is zero only if we are (a) dealing with a linear regression and (b) evaluating that linear regression on the training set. Do you agree? $\endgroup$
    – Adrian
    Mar 30, 2022 at 23:49
  • $\begingroup$ I’m with you so far and am starting to see the connection. Anything else? $\endgroup$
    – Dave
    Mar 31, 2022 at 0:17

1 Answer 1


There seems to be a lack of consistency in what $R^2$ should mean outside of the simple settings.

The simplest case is simple linear regression, where the squared Pearson correlation between the feature and outcome equals the squared Pearson correlation between the true and predicted outcomes. These both equal the "sum of squares formula" you mention from sklearn: $1 - \left[\left(\overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 \right)\Bigg/\left( \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 \right) \right]$. All three of these turn out to be equal to the proportion of variance in $y$ explained by the regression. Thus, in the simple case, there are four notions of what $R^2$ means.

Moving to a more general setting, there might be more than one feature, so the correlation between the feature and outcome no longer makes sense. However, all three other notions can be calculated, and each has a legitimate claim to being called $R^2$. I would go with the "sum of squares" formula, since it has a nice connection with a comparison to a "must beat" model that makes sense to me, but all three notions can be defended. I also like the connection this calculation has to the reduction in error rate that a classifier has (and the fact that the reduction in error rate I discuss at the link relates to a reasonable definition of the familiar $R^2$ statistic makes me like the reduction in error rate statistic all the more).

The good news is that you're always allowed to define a statistic. If you have a reason to want to know the squared correlation between the true and predicted values, feel free to define and calculate such a statistic. If you have a reason to want to use the "sum of squares" formula like sklearn uses, define it and use it. If you want to modify the sklearn formula in the way that makes sense to me for out-of-sample testing, define the formula and use it. If you want to decompose the total sum of squares to get the $SSRes$, $SSReg$, and $Other$ term I discuss here in order to discuss the proportion of the variance in $y$ that is explained by the model, define it and use that formula.

But as far as there being a name that basically every statistician knows, like how I can write $\bar x$ without ambiguity, no, I do not sense that for $R^2$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.