Is it valid using testing data for model selection when validation data cannot do the job?

Question

In Hastie's Elements of statistical learning, it says: "The training set is used to fit the models; the validation set is used to estimate prediction error for model selection; the test set is used for assessment of the generalization error of the final chosen model."

My understanding the authors mean that when using this strategies to compare two algorithms, one should build algorithms on training data and run them on the validation data to compare them and decide on a winner.

I have a further question on this issue: What if validation data (or cross validation) gives Two winners.

Suppose I built two models, say M1 and M2, they perform equally well in cross validation. That is, I do not have a winner using validation data. Now I apply the two models to predict testing data and I see M1 performs better than M2.

My question is: in this case, can I consider M1 as the better model.

This question is basically asking whether it is valid to use testing data to do model selection when validation (cross validaiton) data cannot choose the best model.

Answer 1

I think there's some confusion about the general framework. You should really think of three different sets:

1.) Training data, used to fit each model

2.) Testing data, independent of the training data, which is used to test how the models fit from (1) perform on new data and is used to decide which model appears to work best for predicting new data.

Note that with cross-validation, the the partitioning of (1) and (2) is done repeatedly.

3.) Validation data. In step (2), while we didn't use the testing data to fit the individual models, we did select the model we wanted based on the testing data. Therefore, we did do some fitting with the testing data (i.e., model selection), so we should expect some bias in the performance. To remove this bias, we will withhold some from both the training and testing data, and apply our finally selected model to this hold out. This should give us an unbiased estimate of the average prediction error, under a few assumptions (i.e., the full dataset we are using looks like the data we will use in practice, conditional independence of the samples, etc.).

So with this in mind, I think your question is what to do if the testing data does not appear to show any real difference between two different models. In this case, I would not suggest using your held-out validation set to determine which model to use. The reason for that is that you will have now "poisoned the well", i.e., the whole point of the validation set is that it is supposed to be independent of all modeling decisions so we can get unbiased estimates of predictive error. You lose that once you use it to make modeling decisions. Furthermore, unless the validation set is much larger than the test/training set (which itself is a bad idea), you shouldn't expect to be able to differentiate any better using the validation set.

With that in mind, it appears we have seen that the two models predict pretty evenly, given our current data. In that case, you could consider just randomly picking one of the two (all the evidence so far says they perform equally well), picking one for reasons other than predictive power (i.e., elastic net models are easier to interpret than random forests, so if both preform equally well, chose elastic net model in case you want to dive into the "why" part of your model) or just simply average the predictions from your two models for your final prediction.