There are 2 binary classification models (Denoted modelA and modelB) that we built with different approaches, both of which are expected to output the probability of possitive outcome.

There's a validation set which is temporally separated from the training set.

we want to know which of the models performs better on this validation set.

the validation set is not generous in size.

Using the bootstrap, I calculated the probabilty that the validation AUC (Area Under ROC) of modelA is higher than that of modelB, under the empirical distribution of the validation set.

To be specific, I did the following procedure: 1. bootstrap the validation set a hell lot of times 2. find the proportion of times in which modelA outperforms modelB in terms of AUC.

The result is, modelA has a 11% probability of outperforming modelB.

The questions are: 1. Is this approach proper? 2. If it is, how should I properly interpret this number 11%? 3. If it is not, what is the correct approach to address this kind of question? 4. Does the properness depend on the detailed properties of the statistic (here AUC) ?

p.s.: I read in some paper by Efron (Second thoughts on the bootstrap), that bootstrap has a good property called second-order accuracy, which guarantees the bootstrap variance as a good estimator of variance, whereas the boostrap has no advantage in estimating the mean. This is why I'm used to think twice each time I use bootstrap

To make the concern more specific: Consider $$\mathcal{A} = AUC(modelA) - AUC(modelB)$$ as a statistic on the sample. Then what I'm doing is in fact measuring the probability that $\mathcal{A}$ is greater than 0, which is a measurement of mean value, which I presume bootstrapping isn't good at.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.