0
$\begingroup$

I have a dataset that I separate into training and validation. I fit several models on the training set and then evaluate their performance on the validation set. When I rank the models based on their validation set performance I notice that the ranking is not stable: if I choose half of the validation data points the ranking changes.

Is there a way to calculate the confidence of the resulting rank? Something like bootstrapping the validation set and aggregating the different ranks?

$\endgroup$
  • $\begingroup$ You are thinking of Cross-Validation, since you can apply that as many times as you want and get an estimate with intervals. Can't really do that with one test set, although you maybe shouldn't. Define your test set in advance and use the best result, take it as is. $\endgroup$ – user2974951 Sep 20 '18 at 8:42
  • $\begingroup$ what do you mean that you choose half of the validation data points? $\endgroup$ – JYY Sep 20 '18 at 8:45
  • $\begingroup$ Yi Yang: Say all models are trained on data points x1 to x500 and all are validated on data points x501 to x1000. By choosing half of the validation points I mean I validate on data points x501 to x750 $\endgroup$ – zamazalotta Sep 20 '18 at 12:17
3
$\begingroup$

This is not as uncommon a problem as one might think, a train-test split can result in very unstable results. Frank Harrell has written about the problems with split-sample validation, see this blog post (and its links). He mentions how you need around 20000 observations for split-sample to be reliable even under optimistic assumptions.

Bootstrapping is a more stable alternative, simulations I've done in the past suggest that it's two to four times as efficient as split-sample in simple settings. Your suggested approach sounds a bit ad-hoc (and hence risky); I would suggest you look into the methodology. There are many resources on the right way to do bootstrap validation of methods, the already-mentioned blog post provides some in its links and a search on this site should provide more.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.