I use 70% of the dataset for training and 30% for testing. I use oversampling on the training dataset with an ANN. I use the test dataset on my ANN and look at the performance of oversampling against not using oversampling. I find the best settings for each ANN using oversampling and for the ANN with no oversampling. Each test setting is tried 10 times.
Now I want to see if there is a statistical significance between the best oversampling model and no oversampling model (when looking at mean performance from 10 tests). Which test should I use to do this?
So basically I have these means from my tests and along with that I calculated the standard deviation from the results.
Test dataset is chosen at random, from two populations (710 and 520 000) each at 30%. Each ANN makes a binary classification and I get the AUC score. How can I see if the ANN's ability to predict this population, based on AUC, is statistically significant?