If the test-set RMSE error of a model is less than cross-validated RMSE error, how can I interpret this?

Is this abnormal? Does it imply a mistake in the methodology?

  • $\begingroup$ It doesn't mean anything. It happens. $\endgroup$ – Aksakal Jan 12 '15 at 23:57
  • 1
    $\begingroup$ Did you check confidence intervals on both errors? Are your test and training samples large enough to warrant the conclusion that the test error is lower? $\endgroup$ – cbeleites Jan 13 '15 at 9:33
  • $\begingroup$ Yes, the C.I's are non-overlapping and the test error is lower. Is there something u'd like to infer about this situation? Else, I was planning to accept Aksakal's answer as final..unless there is a different thought put-forth. $\endgroup$ – user75402 Jan 13 '15 at 19:00
  • $\begingroup$ It happens a lot in machine learning. I do not think that it means nothing. I notice a high correlation between this phenomena and poor performance during further out of sample validation where the ground truth is known. It may be a form of overfitting in the case of bagging models, since cross-validation is used to inform the model during training in this case. $\endgroup$ – Hack-R Feb 28 '16 at 20:25

Like Aksakal said this is something that can happen naturally. As an illustration, imagine a dataset X which is split into pieces X_1 and X_2. Suppose the rmse for a model trained on X_1 but tested on X_2 is 2 and the rmse for a model trained on X_2 but tested on X_1 is 3. Then if we used two fold cross validation (with the above folds) we'd find that the CV error would be 5/2, right? Then suppose, separately and by chance, we chose our training set to be X_1 and our test set to be X_2. We would find that our test error would be 2. Nothing was necessarily wrong with our methodology but we ended up with test rmse less than our CV rmse.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.