If the test-set RMSE error of a model is less than cross-validated RMSE error, how can I interpret this?
Is this abnormal? Does it imply a mistake in the methodology?
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
Like Aksakal said this is something that can happen naturally. As an illustration, imagine a dataset X which is split into pieces X_1 and X_2. Suppose the rmse for a model trained on X_1 but tested on X_2 is 2 and the rmse for a model trained on X_2 but tested on X_1 is 3. Then if we used two fold cross validation (with the above folds) we'd find that the CV error would be 5/2, right? Then suppose, separately and by chance, we chose our training set to be X_1 and our test set to be X_2. We would find that our test error would be 2. Nothing was necessarily wrong with our methodology but we ended up with test rmse less than our CV rmse.