I have created two machine learning models and want to run significance tests on the results of various metrics (sensitivity, specificity, Cohen's kappa etc.) to see if there's any statistically significant differences in the results.
My dataset has 500 cases, and of these, a random sampling function chooses 400 cases to train each of the models on and the models are tested on the remaining 100 cases. Therefore, a paired comparison is needed, as the models are tested on the same 100 cases. This process is then repeated 1000 times.
I now have 1000 values for sensitivity etc. for each of the two models - which test should I use to compare them to obtain a p-value?
I've read papers which use Wilcoxon Signed-Rank tests - could I do this?
Does the fact that the random train-test split occurs 1000 times necessitate a statistical correction to be carried out?