I use 70% of the dataset for training and 30% for testing. I use oversampling on the training dataset with an ANN. I use the test dataset on my ANN and look at the performance of oversampling against not using oversampling. I find the best settings for each ANN using oversampling and for the ANN with no oversampling. Each test setting is tried 10 times.

Now I want to see if there is a statistical significance between the best oversampling model and no oversampling model (when looking at mean performance from 10 tests). Which test should I use to do this?

So basically I have these means from my tests and along with that I calculated the standard deviation from the results.

Test dataset is chosen at random, from two populations (710 and 520 000) each at 30%. Each ANN makes a binary classification and I get the AUC score. How can I see if the ANN's ability to predict this population, based on AUC, is statistically significant?

  • $\begingroup$ What do you need this test for? $\endgroup$
    – Tim
    May 17, 2022 at 17:31
  • $\begingroup$ To see if oversampling is significant or not $\endgroup$
    – Don_twice
    May 17, 2022 at 18:54
  • $\begingroup$ What for? Why does it matter? $\endgroup$
    – Tim
    May 18, 2022 at 10:11
  • $\begingroup$ Now I understand. I am completing a paper as part of my studies. I am allowed to get help and use it as long as I declare it as not mine, which I of course intend to do! Based on: ieeexplore.ieee.org/document/6790639/authors#authors it seems as if 5x2 cross validation along with a student t-test seems promising? However, I wonder if oversampling might effect this approximation? $\endgroup$
    – Don_twice
    May 18, 2022 at 12:21
  • $\begingroup$ Pardon me for rushing my answer, I read more than the abstract of my prior link. I think my experiment consists of comparing two classifiers and I have enough data for testing. As such I should perform a McNemar's test. However, the testing data needs to be the same for all classifiers tested. This could add errors since I am not using all data for testing. Perhaps perform k-fold cross validation and McNemar's test? Could this be something useful? $\endgroup$
    – Don_twice
    May 18, 2022 at 12:46


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.