5
$\begingroup$

When calculating the $R^2$ value of a linear regression model, should it be calculated on the training dataset, test dataset or both and why?

Furthermore, when calculating $SS_{\text{res}}$ and $SS_{\text{tot}}$ as per the wikipedia article above, should both sums be over the same data set? In other words, if calculating $SS_{\text{res}}$ over the training dataset, does that require that $SS_{\text{tot}}$ also be calculated over the training dataset? (and similarly for the test dataset.)

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ For the second question, I do not see any reason why you should not calculate both sums over the same dataset. For the first, it will depend on your goals. When you compute R2 on the training data, R2 will tell you something about how much of the variance within your sample is explained by the model, while computing it on the test set tells you something about the predictive quality of your model. $\endgroup$ – Christoph Hanck May 26 '18 at 15:07
  • 1
    $\begingroup$ For all those data sets you are gonna calculate the $R^2$ value (or some similar value) once or multiple times (multiple times for doing optimization and scanning to find some optimal parameter set used in the fitting). But for the different sets it is done for different reasons. See for instance: stats.stackexchange.com/questions/19048/… $\endgroup$ – Sextus Empiricus May 27 '18 at 19:23
6
$\begingroup$

The test data shows you how well your model has generalized. When you run the test data through your model, it is the moment you've been waiting for: is it good enough?

In the machine learning world, it is very common to present both the validation and the test accuracy, but it is the test accuracy that is the most important.

However, if you get a low $R^2$ score on one, and not the other, then something is off! E.g. If the $R^2_{\text{test}}\ll R^2_{\text{training}}$, then it indicates that your model does not generalize well. That is, if e.g. your test set only contains "future" data points, then your model would not appear to extrapolate well (aka a form of covariate shift).

In conclusion: you should compare them! However, in many cases, it's the test-set results you're most interested in.

Improve this answer
$\endgroup$
5
  • $\begingroup$ This is pretty hard to arrange in practice in something like Tensorflow :( $\endgroup$ – rjurney Nov 7 '20 at 17:57
  • 2
    $\begingroup$ What specifically do you find hard about it? Maybe open a separate question about it? $\endgroup$ – Andreas Storvik Strauman Nov 13 '20 at 12:52
  • $\begingroup$ I’ve implemented adjusted R squared for my model as a metric in Tensorflow, but I’m not aware how to pass different metrics for train and test set metrics and it takes the x and y shapes as parameters. I found r squared itself to actually be harmful in modern machine learning with lots of records and features. It tends to be very high even when performance is very poor. $\endgroup$ – rjurney Nov 14 '20 at 20:08
  • $\begingroup$ @rjurney It's important, as with any metric, to be sure you're using it to measure the right thing. If R2 is high, then model performance is high according to, well, the criteria R2 has. If you get a high R2 score on a "low performing model" then I guess it's not the R2 criteria you want to evaluate the model on? If you have a metric that suggests that it is low performing, then that's is the metric you should use, as that's how you ultimately evaluate? R2 gives you exactly what it's designed for, and is not a generalized metric that universally tells you whether your model performs well. $\endgroup$ – Andreas Storvik Strauman Nov 15 '20 at 8:51
  • $\begingroup$ Right, but I never found anything saying, “R Squared isn’t appropriate for lots of data or lots of features.” It should always be presented in that light because that is a very common case in which it doesn’t indicate performance whatsoever. $\endgroup$ – rjurney Nov 18 '20 at 2:27
1
$\begingroup$

When calculating the $R^2$ value of a linear regression model, should it be calculated on the training dataset, test dataset or both and why?

The usual $R^2$ is a fitting measure and must be calculated on the training set. In some regression analysis there is no split in vs out of sample and "in sample = all data".

Furthermore, when calculating $SS_{\text{res}}$ and $SS_{\text{tot}}$ as per the wikipedia article above, should both sums be over the same data set?

Yes, of course.You have three object: residual sum of square, total sum of square, explained sum of square. All of them are computed "in sample".

If you are interested in prediction accuracy, to use in sample measures, as the usual $R^2$, is not a good idea (quite common mistake in the past). You need the so called out of sample $R^2$. Read here: How to calculate out of sample R squared?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.