I have a dataset that I separate into training and validation. I fit several models on the training set and then evaluate their performance on the validation set. When I rank the models based on their validation set performance I notice that the ranking is not stable: if I choose half of the validation data points the ranking changes.
Is there a way to calculate the confidence of the resulting rank? Something like bootstrapping the validation set and aggregating the different ranks?