# Using $R^2$ to evaluate out-of-sample performance

In this paper the $R^2$ is used to evaluate out-of-sample predictions for several methods including neural networks and tree based methods (see section 3.3 Evaluation and Validation).

How is the out-of-sample $R^2$ computed and is it a valid performance measure?

Edit: The link did not work. I fixed it now.

• I calculate R-squared (R2) as "R2 = 1.0 - (absolute_error_variance / dependent_data_variance), and this tells me what fraction of the dependent data variance is explained by the model. If for example the R2 value is 0.95, the model explains 95% of the dependent data variance. – James Phillips May 21 '18 at 17:08

The coefficient of determination, $R^2$, value can be calculated as $$R^2 = 1 - SS_{res}/SS_{tot}$$ where the total sum of squares is $$SS_{tot} = \sum_{i}(y_i - \bar{y})^2$$ and the residual sum of squares is $$SS_{res} = \sum_{i}(y_i-\hat{y_i})^2 = \sum_{i}e_i^2$$ To apply the above equations to out-of-sample predictions you could use $y_i$ and mean $\bar{y}$ from your test data. This seems like the most obvious way of calculating out-of-sample $R^2$.
If the model prediction is better than simply assuming a constant fit equal to the mean, then the $R^2$ will be greater than zero. As $SS_{res}$ goes to zero, the $R^2$ would approach one. Not that mean squared prediction error (MSPE) would also go to zero as $SS_{res}$ goes to zero. Therefore, if there is low MSPE we would expect high values of $R^2$. The $R^2$ value can also be interpreted as one minus the ratio of MSPE to variance.
$$R^2 = 1 - MSPE/Var(y) = 1 - \frac{\frac{1}{N}\sum_{i}(y_i-\hat{y_i})^2}{\frac{1}{N}\sum_{i}(y_i-\bar{y})^2}$$
One of the main criticisms of $R^2$, that would apply to out-of-sample as well, is that if the data is very noisy the $R^2$ may be low even if the model fits the data well. Also, the $R^2$ could be high even if the functional form of the model is different than that of process that generated the data (e.g., a linear fit to a quadratic function).
• When using the $R^2$ this way it looses its interpretation of the ratio of explained variance to the total variance. It is also not bounded to the $[0,1]$ interval. Somehow I cannot see why people would use it instead of e.g. MSE. Is there anything I am missing? – Alex May 22 '18 at 6:46
• I believe normal $R^2$ can also be negative. For example, this could happen if you do not have a constant term in linear regression. I think we might also be able to interpret it as ratio of explained variance out of sample to total variance out of sample. I think one benefit of approach is that it puts MSPE in perspective with respect to variance of data. – Nat May 22 '18 at 23:39