# How to calculate out of sample R squared?

I know this probably has been discussed somewhere else, but I have not been able to find an explicit answer. I am trying to use the formula $R^2 = 1 - SSR/SST$ to calculate out-of-sample $R^2$ of a linear regression model, where $SSR$ is the sum of squared residuals and $SST$ is the total sum of squares. For the training set, it is clear that

$$SST = \Sigma (y - \bar{y}_{train})^2$$

What about the testing set? Should I keep using $\bar{y}_{train}$ for out of sample $y$, or use $\bar{y}_{test}$ instead?

I found that if I use $\bar{y}_{test}$, the resulting $R^2$ can be negative sometimes. This is consistent with the description of sklearn's r2_score() function, where they used $\bar{y}_{test}$ (which is also used by their linear_model's score() function for testing samples). They state that "a constant model that always predicts the expected value of y, disregarding the input features, would get a R^2 score of 0.0."

However, in other places people have used $\bar{y}_{train}$ like here and here (the second answer by dmi3kno). So I was wondering which makes more sense? Any comment will be greatly appreciated!

First of all is need to say that for prediction evaluation, then out of sample, the usual $$R^2$$ is not adequate. It is so because the usual $$R^2$$ is computed on residuals, that are in sample quantities.

We can define: $$R^2 = 1 – RSS/TSS$$

RSS = residual sum of square

TSS = total sum of square

The main problem here is that residuals are not a good proxy for forecast errors because in residuals the same data would be used for both, model estimation and model prediction accuracy. If residuals (RSS) are used the prediction accuracy would be overstated; probably overfitting occur. Even TSS is not adequate as we see later. However we have to say that in the past the mistaken use of standard $$R^2$$ for forecast evaluation was quite common.

The out of sample $$R^2$$ ($$R_{oos}^2$$) maintain the idea of usual $$R^2$$ but in place of RSS is used the out of sample MSE of the model under analysis (MSE_m). In place of TSS is used the the out of sample MSE of one benchmark model (MSE_bmk).

$$R_{oos}^2 = 1 – MSE_m/MSE_{bmk}$$

One notable difference between $$R^2$$ and $$R_{oos}^2$$ is that

$$0 \leq R^2 \leq 1$$ (if the constant term is included)

while $$-\infty \leq R_{oos}^2 \leq 1$$

If $$R_{oos}^2 < = > 0$$ the competing model perform worse/equal/better than the benchmark one. If $$R_{oos}^2 =1$$ the competing model predict perfectly the (new) data.

Here we have to keep in mind that the even for the benchmark model we have to consider the out of sample performance. Therefore the variance of the out of sample data underestimate $$MSE_{bmk}$$.

In this sense something like $$MSE_{bmk} = (1/n)\Sigma (y - \bar{y}_{test})^2$$ seems to me a wrong choice. While something like $$MSE_{bmk} = (1/n)\Sigma (y - \bar{y}_{train})^2$$ seems to me plausible.

In my knowledge this measure was proposed for the first time in: Predicting excess stock returns out of sample: Can anything beat the historical average? - Campbell and Thompson (2008) - Review of Financial Studies. In it the the bmk forecast is based on the prevailing mean given information at time of the forecast.

• The Campbell/Thompson argument would seem to favor using $\bar y_{train}$, agreed?
– Dave
Mar 22 at 14:30
• It seems me that Campbell and Thompson do not speak, at least not exsplicitely, about this question. However, what matter for oosR-squared is that both predictions of the model under analysis and of benchmark model can be computed with data actually available at the time of the prediction. So someting like $Y_{test}$ sound me a wrong choice. Campbell and Thompson benchmark forecast are not a constant (a plain average) but are consistent with previus concept. May 2 at 16:42
• "predictions...of benchmark model can be computed with data actually available at the time of the prediction" This is the logic behind my stance, and I am with you that this is consistent with the way Campbell and Thompson think of the problem.
– Dave
May 2 at 16:53

You are correct.

The OSR$^2$ residuals are based on testing data, but the baseline should still be training data. With that said, your SST is $SST=Σ(y−\bar y_{train})^2$; notice that the is the same for $R^2$

• Although I have fixed some obvious and some apparent errors from previous edits some of the notation and some of the intended meaning are still unclear. Sep 27, 2017 at 20:27
• Thanks for the answer! Do you have any reference on this? It seems stat softwares use commonly the alternative definition, with y_test? May 1, 2019 at 1:53
• Do you have a reference for this? Granted, if you take $R^2$ to be a comparison of deviances, ergo a comparison of likelihoods I think you're right. But if you take $R^2$ to be the proportion of explained variance then not, because the total sum of squares won't appear anywhere. May 12, 2020 at 21:44
• @Firebug I wouldn’t worry about the interpretation of $R^2$ as proportion of variance explained. That turns out to be the exception, not the rule. Further, we don’t even need a nonlinear regression for that interpretation to break down.
– Dave
Mar 29, 2022 at 10:36

We have just published an article on this subject in The American Statistician here

Similar to @markowitz, we define out-of-sample $$R^2$$ as a comparison of two out-of-sample models: the null model using only the mean outcome of the training data $$\bar{y}_{train}$$, and the more elaborate model using covariate information.

For the squared error loss of the null model (which we call the MST), we derive an analytical expression showing that

$$MST = \operatorname{Var}(\bar{Y}_{train}) + \operatorname{Var}(Y) = \frac{n+1}{n}\operatorname{Var}(Y),$$

meaning that the prediction error is a sum of estimation error on $$\bar{y}_{train}$$ and irreducible error. This is a useful expression in absence of a test set. But if you have an independent test set, I would indeed prefer the expression $$n^{-1}\sum_{i \in \text{test}}(y_i-\bar{y}_{train})$$ as suggested above. In principle, both estimators have the same estimand, but the latter is more robust to differences between training and test sets. Finally, we show through simulation that the expression $$n^{-1}\sum_{i \in \text{test}}(y_i-\bar{y}_{test})$$ can be badly biased for estimation the true $$R^2$$.

The squared error loss of the elaborate model (the MSE) is then to be estimated through cross-validation or on your test set. Corresponding out-of-sample $$R^2$$ is then simply

$$\hat{R}^2 = 1-\frac{\widehat{MSE}}{\widehat{MST}}$$

We provide a standard error for this estimate, unlocking hypothesis testing and confidence intervals.

REFERENCE

Hawinkel, Stijn, Willem Waegeman, and Steven Maere. "Out-of-sample R 2: estimation and inference." The American Statistician just-accepted (2023): 1-16.

• This is awesome. I disagree with the interpretation of explained variance, but this is a minor point. Your article makes the exact kind of argument I make all the time on here. It is good to see that in a legitimate journal. If I ever get around to writing an article I have in mind, I will cite your work.
– Dave
May 26 at 10:16
• @Dave thanks for your kind words! In your derivation, if $y_i$ is the out-of-sample observation and $\bar{y}$ and $\hat{y}_i$ are calculated based on in-sample data (and out-of-sample regressors), I am not sure that your "Other" term equals zero in the general case of a random design. The proofs I find are for in-sample linear regression, which is the subject of your post, but aren't matters different for out-of-sample prediction? If the out-of-sample design is fixed, I think this term drops for any unbiased prediction model, not just for linear models. May 30 at 14:13
• That is my point: that "other" term is not zero in general, and then claims of $R^2$ as being the proportion of variance explained become dubious.
– Dave
May 30 at 23:43