In Machine Learning the standard practice for learning a Classifier --e.g. fitting a Logistic Regression model-- and then validating its performance is to split the original/available Dataset into a train and test dataset randomly --typically 70% of the data used for training and 30% for validation. Using the training dataset you fit a model -- possibly using k-fold cross-validation on the training dataset -- and then make predictions on the Test (Unseen / Out of Sample) dataset. You measure the performance of your predictions on the Test dataset and your report them. That's it.
The problem with this approach is that the particular way you split the original dataset into Training and Test datasets may influence the performance of your model and may lead to wrong conclusions regarding the best algorithm for the data at hand and the best hyper-parameter values.
You can test that by repeatedly splitting randomly a dataset into training and test -- you will observe differences in the performance of the algorithms / hyper-parameter settings.
And while k-fold cross validation should alleviate the problem of using a particular subset of the original dataset for training purposes, it does nothing to mitigate the bias introduced by a particular choice of the Test dataset (from all the possible combinations).
I was thinking that one way to deal with this problem would be to split randomly the original dataset 1,000 times into Train and Test, fit each time a model on the given Train dataset, evaluate its performance on its complement Test dataset, record the performance and continue. In the end report the distribution of the performance measure (e.g. AUC), i.e. mean and standard deviation.
I find this idea to have an analogy to the Bootstrapping method, in the sense you resample 1,000 times the train and test datasets from the same initial sample in a random way -- although not with replacement.
Bootstrapping theory claims that the distribution of the Estimator can be generalized to the Population. Could we make an analogous claim for the distribution of the Estimator of the Performance metric we gauge in each split?
Your advice will be appreciated.