1
$\begingroup$

I am running linear regression, I have over 80,000 observations and 5 predictor variables.

I am trying to pick the best model with RMSE and R-squared and I understand how all that works but...

What is a significant difference between RMSE's ? For example, one model has RMSE = 2664.02 and the other has RMSE = 2665.13

What do I do? They are literally the same. Do I run the model with less variables? R-square values are almost identical also. What would be a significant RMSE difference?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I assume your RMSEs are calculated on holdout samples, correct?

You will need to get a handle on the variability of the RMSE. I would recommend a mixture between a bootstrap and cross validation: randomly pick 80% of your data, fit both models to these samples, use the fitted models to predict the remaining 20%, note both RMSE. Repeat many (e.g., 1,000) times. Assess the overlap between the two RMSE distributions, or conduct a paired t-test.

And of course, statistical difference is not the same as practical difference, especially in a case such as this one.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.