I have two models $A$ and $B$ as well as a training set $X$. I want to test whether there is a significant difference in performance between model $A$ and $B$.
I'm attempting to do this via bootstrapping, i.e.
Draw a bootstrap sample $X*$
Train model $A$ and $B$ on $X*$ to get model $A*$ and $B*$
Obtain a single scalar performance estimate for $A*$ and $B*$ by testing on the out-of-bootstrap samples
Repeat this $B$ times
What is the correct way to compare the obtain performance distributions?