Often deep learning/machine learning models are compared just by measuring their performance on a validation dataset - the model that performs better "wins".
Within null hypothesis testing, two samples are "truly" different only if the discrepancy is large enough to be "unlikely" if the null hypothesis were true.
How can I determine if the improvement is real rather than spurious? Can I just use bootstrap to calculate the CI for the accuracy (or mAP, or L2 norm, etc.), or are there better (more robust/sensitive) methods?
Edit: I'm specifically talking about deep learning models. It often takes days to train a DL model, so k-fold cross-validation (or really anything that requires to re-train the model) isn't a practical option.
What are the best practices to control for the effect of multiple comparisons? [e.g. if I try to beat the performance of
Fon the dataset
X, sooner or later I will succeed - how can I avoid doing that for the wrong reason?]
Are there practices that "help" not getting stuck during manual hyperparameter search? [e.g. after having found the optimal LR, number of layers/filters, etc. for a given optimization algorithm (e.g. SGD) if I replace it with a new one (e.g. Adam) I think might observe a drop in the performance even if the latter was a better choice, because of the co-dependence between the different hyperparameter]