# Bootstrap Hypothesis Test: Comparing the Performance of two Models

I have two models $$A$$ and $$B$$ as well as a training set $$X$$. I want to test whether there is a significant difference in performance between model $$A$$ and $$B$$.

I'm attempting to do this via bootstrapping, i.e.

1. Draw a bootstrap sample $$X*$$

2. Train model $$A$$ and $$B$$ on $$X*$$ to get model $$A*$$ and $$B*$$

3. Obtain a single scalar performance estimate for $$A*$$ and $$B*$$ by testing on the out-of-bootstrap samples

4. Repeat this $$B$$ times

What is the correct way to compare the obtain performance distributions?

• This would be one way od doing it, but why are you not using Cross Validation? – user2974951 Jan 10 at 8:16
• @user2974951 I was following the bootstrap method in chapter 7 of the Elemtents of Statistical Learning. How would I estimate significance using CV? – bi_scholar Jan 10 at 10:04