The conventional definition of $R^2$ is: $R^2 = 1-SSE/SST$, where SSE denotes sum of squared errors and SST is total sum of squares ($n\times variance$, n being number of sample points in train set).

However, I want to see the fitness of my model on out-of-sample set (test set) or both train and test sets combined. Is it fine to use the same definition of $R^2$ by taking SSE and SST appropriately over test set or (train+test) sets, respectively? Eg. I train my model on $n$ sample points. I want to check its performance on ($n+p$) points ($p$ being some new sample points out of the train set). Can I use $R^2 = 1-(SSE$ on (n+p) points$)/(SST$ on (n+p) points$)$?


3 Answers 3


It makes sense. It is more common however to keep the training and test sets separated. So that you train your model on the train set, and then predict on the test set alone. From there you can calculate the prediction error, and a $R^2_{pred}$ if you like. (train on $n$ data points, evaluate on $p$ data points, in your terms.)

You can also look up stuff like the PRESS statistic, and other cross validation methods.

  • 2
    $\begingroup$ It's especially recommended to keep separate training and test (and validation) sets to avoid overfitting and overoptimistic measure of model fit. $\endgroup$
    – chl
    Oct 17, 2020 at 18:42

It makes sense to apply $R^2= 1-{\sum(y_i-\hat y_i)^2}/{\sum(y_i-\bar y)^2}$ to test set directly. It's a measure of the size of squared residuals compared to the variance of true values.

Alternatively, if you adopt the notion of deviance (see this answer), then you might use the null model from training data instead:

$$\tilde R^2= 1-\frac{\sum(y_i-\hat y_i)^2}{\sum(y_i-\bar y_{train})^2}$$

I've seen both in use, and both can be justified.

  • $\begingroup$ I cannot think of a justification for using $\bar y$ instead of $\bar y_{train}$. What have you seen? $\endgroup$
    – Dave
    May 2 at 20:21
  • $\begingroup$ @Dave And yet it's form used in almost every package and paper I ever read. It's how it's implemented in sklearn, pytorch and TF2 for example $\endgroup$
    – Firebug
    May 3 at 7:41
  • $\begingroup$ I believe a justification lies in the fact that $\bar{y}_{test}$ is an estimator of $\bar{y}_{train}$ for most usual distributions, but for small test sets the variance plays an important part. I prefer the second definition, since it's a clearer and meaningful comparison, but it hasn't catch on. $\endgroup$
    – Firebug
    May 3 at 7:44

There are several definitions of $R^2$ that are equivalent for in-sample OLS linear regression.

  1. The correlation between the $x$ and $y$ in simple linear regression

  2. The correlation between the predictions $\hat y$ and truth $y$

  3. The proportion of variance explained

  4. A comparison of the model performance to the performance of a naïve model that always predicts $\bar y$, no matter what values the features take

The final one is the one that makes the most sense to me.

When you get to out-of-sample $R^2$, the first three present issues.

  1. Typically, an out-of-sample metric is of interest to problems that require more complicated feature spaces than just one variable, so this is out.

  2. Out-of-sample, all bets are off. If you've badly overfit, you could be in a position where high values of $y$ correspond to low values of $\hat y$, and low values of $y$ correspond to high values of $\hat y$, the extreme of which is $\hat y = -y$. This puts you in a position where $cor(y, \hat y)<0$, and when you square that value, you miss that the predictions are terrible. Further, this approach misses systematic bias (consistently predicting too high or too low by $k$) and predicting multiples of the true values. That is, if $\hat y = a+b y$ for $(a,b)\ne(0,1)$, $cor(y, \hat y)$ is oblivious to the poor predictions. That this metric misses such critical information is, to me, a dealbreaker.

  3. This one is appealing, but the "proportion of variance explained" interpretation of $R^2$ breaks down in most situations. Out-of-sample is one such situation, as the coefficients that result in the orthogonality needed for this interpretation to hold out-of-sample are unlikely to be the coefficients estimated in-sample, even for an OLS linear regression.

  4. Finally, this one makes sense. We have a model and are interested in the square loss. As minimizing square loss corresponds to finding the conditional mean, a good benchmark is to see if the predictions are better than a model that always predicts the conditional mean to be the pooled mean.

$$ \dfrac{ \sum_{i=1}^n\big( y_i - \hat y_i \big)^2 }{ \sum_{i=1}^n\big( y_i - \bar y \big)^2 } $$

If the numerator is smaller than the denominator, it means that our predictions beat the predictions made by out baseline model that naïvely predicts $\bar y$ every time. If the numerator is larger than the denominator, then all of our statistics and machine learning efforts are doing worse than we would do by predicting AVERAGE(A:A) (to use some Excel terminology). That is, our model is doing a poor job of predicting.

It is typical to subtract this quantity from $1$ in order to align with the other three ways of defining in-sample $R^2$.

$$R^2 = 1- \dfrac{ \sum_{i=1}^n\big( y_i - \hat y_i \big)^2 }{ \sum_{i=1}^n\big( y_i - \bar y \big)^2 } $$

This idea of comparing to a baseline model exists for other metrics. For logistic regressions, there are two named metrics that do exactly this: Efron's and McFadden's pseudo $R^2$, as discussed on this UCLA page.

For evaluating an out-of-sample $R^2$, I would use the following:

$$R^2_{oos}=1- \dfrac{ \sum_{i=1}^n\big( y_i - \hat y_i \big)^2 }{ \sum_{i=1}^n\big( y_i - \bar y_{train} \big)^2 } $$

This compares the out-of-sample performance of your model to the out-of-sample performance you would get from a model trained just to predict the mean every time (the naïve baseline model).

Irritatingly, the popular Python machine learning package sklearn has an out-of-sample $R^2$ function, sklearn.metrics.r2_score, that uses the $\bar y$ from whatever you input as the truth values. This is fine for in-sample $R^2$, but for out-of-sample $R^2$, it results in the following formula:

$$R^2_{oos}1- \dfrac{ \sum_{i=1}^n\big( y_i - \hat y_i \big)^2 }{ \sum_{i=1}^n\big( y_i - \bar y_{test} \big)^2 } $$

The denominator is now based on the square loss of an intercept-only linear model that has been trained on the test data. We should never have access to this model, as it requires us to train on the test data, and I disagree with the sklearn implementation. Fortunately, however, this function does not fall for the traps that just the $cor(y, \hat y)$ does.

from sklearn.metrics import r2_score
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt


N = 100
y = np.random.uniform(0, 1, N)

yhat1 = y + 4
plt.scatter(y, yhat1)
print(r2_score(y, yhat1))
print(np.corrcoef(y, yhat1)**2)

yhat2 = y * 3
plt.scatter(y, yhat2)
print(r2_score(y, yhat2))
print(np.corrcoef(y, yhat2)**2)

yhat3 = -y
plt.scatter(y, yhat3)
print(r2_score(y, yhat3))
print(np.corrcoef(y, yhat3)**2)

yhat4 = 2 + 3*y
plt.scatter(y, yhat4)
print(r2_score(y, yhat4))
print(np.corrcoef(y, yhat4)**2)

In all of these situations, the squared correlation between the predictions and the truth is a perfect-looking $1$, yet the sklearn.metrics.r2_score correctly indicates that the predictions are terrible.

Finally, for evaluating $R^2$ on the combined data (training and testing), it is unclear what this would tell you. If I had to do that calculation, I would be inclined to use the in-sample $\bar y$ in the denominator and just do the sums over all $n+p$ values. You might also be interested in bootstrap validation that uses all of the observations.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.