0
$\begingroup$

I often run into the situation where I have several regression models, each of which gives a RMSECV, and I need to choose which one is "best". Of course, I can choose the one with the minimum RMSECV, but sometimes several models have very similar values. What is the correct way to determine whether a difference in RMSECV is meaningful? Or conversely, how do I tell when two regression models have RMSECVs that are close enough that they should be considered statistically equivalent?

$\endgroup$
  • $\begingroup$ I would argue that's a subject matter decision and not a statistical decision. Also, with the same data and associated results, one model might have a slightly higher RMSE but is either much cheaper to implement than a model with more expensively obtained predictors. And there are many times political considerations which might likely change over time. So, in short, there are no correct answers but probably reasonable answers for which you might find a consensus. $\endgroup$ – JimB Jan 11 '18 at 18:47
  • $\begingroup$ rdocumentation.org/packages/forecast/versions/8.1/topics/… $\endgroup$ – Christoph Hanck Jan 12 '18 at 7:16
0
$\begingroup$

If you are using a regression method that is not complex and not stochastic, you should have always a fixed performance value for each attempt. Therefore, the higher -> the winner. However, if you are using for example ANNs, then you always have a random component in your results. If this is your case, you might want to know if the difference in the results come from this random component or if indeed you have actually found a better (or worse) predictor.

To make sure about this, you can use repeated cross validation a good amount of times for each predictor. After that, you just need to compare using hypothesis tests if the differences are significant enough or not.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.