Split the validation set or leave it be?

Question

Suppose we want to select one of $m$ models $M_1,\cdots, M_m$ based on their performance on a validation set $S$. We could do this in several different ways:

1. Pick the model $M_{i_1}$ that minimizes empirical loss $\text{EL}(M_i,S)$
2. (Randomly) split the validation set into $m$ equal-sized validation sets $S_1,\cdots, S_m$ and choose that model $M_{i_2}$ which has the smallest $\text{EL}(M_i,S_i)$

Is one of the above methods preferable to the other?

Can anything be said about $\text{EL}(M_{i_1},S)$ vs $\text{EL}(M_{i_2},S_{i_2})$, i.e. how the empirical losses of the chosen models compare?

Answer 1

You should be precise about what you mean by validation set and how it has been generated and used.

I'm going to assume you have a standard train, test, validate split on your data, and that you've made use of your training and test data to develop a set $M$ of $i$ processes that build models.

If this is the case, you'll want to use your complete validation set $S$ to estimate performance for every element of $M$; any subset of $S$ is going to be a less powerful estimate of the generalisation performance of $M_i$.

The notation you've used also has me wondering whether you're suggesting testing each $M_i$ on its corresponding $S_i$ only? This has other problems, because it would not only be an under-powered estimate of generalisation error, but each model would be evaluated on a different subset of data. Smaller $S$ and larger $m$ would reduce the stability of your loss estimate. You could explore the extent of this effect by reshuffling or resampling the split of $S$

Finally, I'd recommend reading up on cross-validation and nested cross-validation as an alternative to data splitting. If you have huge amounts of data it may not be necessary, but I think understanding these topics will help you answer your question.