I have two models $A$ and $B$ as well as a training set $X$. I want to test whether there is a significant difference in performance between model $A$ and $B$.

I'm attempting to do this via bootstrapping, i.e.

  1. Draw a bootstrap sample $X*$

  2. Train model $A$ and $B$ on $X*$ to get model $A*$ and $B*$

  3. Obtain a single scalar performance estimate for $A*$ and $B*$ by testing on the out-of-bootstrap samples

  4. Repeat this $B$ times

What is the correct way to compare the obtain performance distributions?

  • $\begingroup$ This would be one way od doing it, but why are you not using Cross Validation? $\endgroup$ – user2974951 Jan 10 at 8:16
  • $\begingroup$ @user2974951 I was following the bootstrap method in chapter 7 of the Elemtents of Statistical Learning. How would I estimate significance using CV? $\endgroup$ – bi_scholar Jan 10 at 10:04

What I think the authors are suggesting (I haven't checked) is to repeatedly train models A and B on bootstrap samples and then compare the two statistics using a test (for ex. proportion test if the statistic is between 0 and 1) and from the test draw a p-value. With a similar approach you could use say 5-fold CV to repeatedly train models A and B on random subsamples of the data and then compare the two statistics. The permutation test would be able to capture more variability in the data (I assume, usually why it's done) but it is going to be more computationally intensive.

  • $\begingroup$ Thanks for your answer, however the authors suggest exactly what I described above as a procedure to estimate the performance mean and variance of a model. Intuitively, this allows for comparison via a significance test but this part is spared out in the book. Could you elaborate a bit more how you would use CV in that context? It seems to me that you would only obtain 5 performance values for both model A and B, which seems insufficient for estimating the performance distribution. $\endgroup$ – bi_scholar Jan 10 at 11:09
  • $\begingroup$ The procedure that I described is exactly what you presented, I only added the statistical test at the end (which is what I assumed they were doing, a proportion test). As I mentioned, you would do repeated 5-fold CV, so randomly partition the data into 5 folds, build the models on all folds, then repeat the process by further partitioning the data into 5 more random partitions, build the models, and so on... $\endgroup$ – user2974951 Jan 10 at 11:22
  • $\begingroup$ I see, thanks. Initially I was using a Wilcox test to compare the two distributions, but I didn't find any literature justifying this approach. I'm not sure about the CV part though as (1) it seems more computationally intense, since at every iteration it need to build 5 models and (2) the obtained performance estimates from every CV are very correlated. For example, in the limit of k -> n, the variance of the CV estimates goes to zero. $\endgroup$ – bi_scholar Jan 10 at 12:28
  • $\begingroup$ There isn't much literature since this is rarely done. It's true that the CV approach is not great, permutations are better overall since they will have more variability. As for computational intensity, it depends how many resamples your perform. $\endgroup$ – user2974951 Jan 10 at 12:33
  • $\begingroup$ What is instead the common approach to evaluate whether the performance of two models is significantly different with respect to some metric p? $\endgroup$ – bi_scholar Jan 10 at 12:38

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.