# Statistical test when comparing oversampling to no oversampling on ANN

I use 70% of the dataset for training and 30% for testing. I use oversampling on the training dataset with an ANN. I use the test dataset on my ANN and look at the performance of oversampling against not using oversampling. I find the best settings for each ANN using oversampling and for the ANN with no oversampling. Each test setting is tried 10 times.

Now I want to see if there is a statistical significance between the best oversampling model and no oversampling model (when looking at mean performance from 10 tests). Which test should I use to do this?

So basically I have these means from my tests and along with that I calculated the standard deviation from the results.

Test dataset is chosen at random, from two populations (710 and 520 000) each at 30%. Each ANN makes a binary classification and I get the AUC score. How can I see if the ANN's ability to predict this population, based on AUC, is statistically significant?

• What do you need this test for?
– Tim
May 17, 2022 at 17:31
• To see if oversampling is significant or not May 17, 2022 at 18:54
• What for? Why does it matter?
– Tim
May 18, 2022 at 10:11
• Now I understand. I am completing a paper as part of my studies. I am allowed to get help and use it as long as I declare it as not mine, which I of course intend to do! Based on: ieeexplore.ieee.org/document/6790639/authors#authors it seems as if 5x2 cross validation along with a student t-test seems promising? However, I wonder if oversampling might effect this approximation? May 18, 2022 at 12:21
• Pardon me for rushing my answer, I read more than the abstract of my prior link. I think my experiment consists of comparing two classifiers and I have enough data for testing. As such I should perform a McNemar's test. However, the testing data needs to be the same for all classifiers tested. This could add errors since I am not using all data for testing. Perhaps perform k-fold cross validation and McNemar's test? Could this be something useful? May 18, 2022 at 12:46