1
$\begingroup$

I am doing linear regression on a dataset. I divided the data into training (70%) and testing(30%). Here are the metrics for training and testing data:

Training data: R2 is 0.85 and RMSE is 2339

Testing data: R2 is 0.67 and RMSE is 2238

Based on RMSE, the model does better on the Testing data as it has lower RMSE values. But based on R-square the model performs better in training data. I was hoping that if the model gives low RMSE then it should have a high r-square and vice-versa. Please could anyone explain these conflicting results?

Edit based on whuber's comment: The variance of responses in the training set is higher, so I think that explains the high value of $R^2$ in the training test. So based on this, can we say that $R^2$ can be misguiding sometimes and better to stick with RMSE?

Edit based on Dave's comment: $R^2=1-\frac{\sum(y_{test}-\hat{y}_{test})^2} {\sum(y_{test}-\bar{y}_{test})^2}$ The python code for this:

y_pred_test=model.predict(x_test) ##model is trained model on training data
r2_score(y_test,y_pred_test)
$\endgroup$
9
  • 3
    $\begingroup$ Pay attention to the variances of the responses in the two datasets. $\endgroup$
    – whuber
    Oct 6, 2021 at 13:10
  • $\begingroup$ How do you calculate out-of-sample $R^2$? $\endgroup$
    – Dave
    Oct 6, 2021 at 13:11
  • $\begingroup$ Yes @whuber, the variances of responses are higher in the training set. I also edited the question based on this information. $\endgroup$
    – Kim
    Oct 6, 2021 at 13:27
  • 1
    $\begingroup$ Based on your edit: if you care about prediction accuracy then choose the relevant metric like RMSE to worry about in cross validation. As you found R squared can be extremely misguiding and should only ever be referenced but not used as the end-all-be-all. $\endgroup$
    – Tylerr
    Oct 6, 2021 at 14:02
  • 1
    $\begingroup$ You should be using $\bar y_{train}$ in the denominator. Remember that $R^2$ is a comparison of model performance to performance of a model that naïvely guesses the mean every time. It only makes sense for that mean to be the in-sample (training) mean. $\endgroup$
    – Dave
    Oct 6, 2021 at 14:14

1 Answer 1

1
$\begingroup$

You should be using $\bar y_{train}$ in the denominator, not $\bar y_{test}$. Remember that $R^2$ is a comparison of model performance to performance of a model that naïvely guesses the mean every time. It only makes sense for that mean to be the in-sample (training) mean.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.