3
$\begingroup$

In this paper the $R^2$ is used to evaluate out-of-sample predictions for several methods including neural networks and tree based methods (see section 3.3 Evaluation and Validation).

How is the out-of-sample $R^2$ computed and is it a valid performance measure?

Edit: The link did not work. I fixed it now.

$\endgroup$
  • 1
    $\begingroup$ I calculate R-squared (R2) as "R2 = 1.0 - (absolute_error_variance / dependent_data_variance), and this tells me what fraction of the dependent data variance is explained by the model. If for example the R2 value is 0.95, the model explains 95% of the dependent data variance. $\endgroup$ – James Phillips May 21 '18 at 17:08
1
$\begingroup$

The coefficient of determination, $R^2$, value can be calculated as $$R^2 = 1 - SS_{res}/SS_{tot}$$ where the total sum of squares is $$SS_{tot} = \sum_{i}(y_i - \bar{y})^2$$ and the residual sum of squares is $$SS_{res} = \sum_{i}(y_i-\hat{y_i})^2 = \sum_{i}e_i^2$$ To apply the above equations to out-of-sample predictions you could use $y_i$ and mean $\bar{y}$ from your test data. This seems like the most obvious way of calculating out-of-sample $R^2$.

If the model prediction is better than simply assuming a constant fit equal to the mean, then the $R^2$ will be greater than zero. As $SS_{res}$ goes to zero, the $R^2$ would approach one. Not that mean squared prediction error (MSPE) would also go to zero as $SS_{res}$ goes to zero. Therefore, if there is low MSPE we would expect high values of $R^2$. The $R^2$ value can also be interpreted as one minus the ratio of MSPE to variance.

$$ R^2 = 1 - MSPE/Var(y) = 1 - \frac{\frac{1}{N}\sum_{i}(y_i-\hat{y_i})^2}{\frac{1}{N}\sum_{i}(y_i-\bar{y})^2} $$

One of the main criticisms of $R^2$, that would apply to out-of-sample as well, is that if the data is very noisy the $R^2$ may be low even if the model fits the data well. Also, the $R^2$ could be high even if the functional form of the model is different than that of process that generated the data (e.g., a linear fit to a quadratic function).

$\endgroup$
  • 1
    $\begingroup$ When using the $R^2$ this way it looses its interpretation of the ratio of explained variance to the total variance. It is also not bounded to the $[0,1]$ interval. Somehow I cannot see why people would use it instead of e.g. MSE. Is there anything I am missing? $\endgroup$ – Alex May 22 '18 at 6:46
  • $\begingroup$ I believe normal $R^2$ can also be negative. For example, this could happen if you do not have a constant term in linear regression. I think we might also be able to interpret it as ratio of explained variance out of sample to total variance out of sample. I think one benefit of approach is that it puts MSPE in perspective with respect to variance of data. $\endgroup$ – Nat May 22 '18 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.