I often run into the situation where I have several regression models, each of which gives a RMSECV, and I need to choose which one is "best". Of course, I can choose the one with the minimum RMSECV, but sometimes several models have very similar values. What is the correct way to determine whether a difference in RMSECV is meaningful? Or conversely, how do I tell when two regression models have RMSECVs that are close enough that they should be considered statistically equivalent?
If you are using a regression method that is not complex and not stochastic, you should have always a fixed performance value for each attempt. Therefore, the higher -> the winner. However, if you are using for example ANNs, then you always have a random component in your results. If this is your case, you might want to know if the difference in the results come from this random component or if indeed you have actually found a better (or worse) predictor.
To make sure about this, you can use repeated cross validation a good amount of times for each predictor. After that, you just need to compare using hypothesis tests if the differences are significant enough or not.