I have a dataset that I separate into training and validation. I fit several models on the training set and then evaluate their performance on the validation set. When I rank the models based on their validation set performance I notice that the ranking is not stable: if I choose half of the validation data points the ranking changes.
Is there a way to calculate the confidence of the resulting rank? Something like bootstrapping the validation set and aggregating the different ranks?
This is not as uncommon a problem as one might think, a train-test split can result in very unstable results. Frank Harrell has written about the problems with split-sample validation, see this blog post (and its links). He mentions how you need around 20000 observations for split-sample to be reliable even under optimistic assumptions.
Bootstrapping is a more stable alternative, simulations I've done in the past suggest that it's two to four times as efficient as split-sample in simple settings. Your suggested approach sounds a bit ad-hoc (and hence risky); I would suggest you look into the methodology. There are many resources on the right way to do bootstrap validation of methods, the already-mentioned blog post provides some in its links and a search on this site should provide more.
