0
$\begingroup$

I use 70% of the dataset for training and 30% for testing. I use oversampling on the training dataset with an ANN. I use the test dataset on my ANN and look at the performance of oversampling against not using oversampling. I find the best settings for each ANN using oversampling and for the ANN with no oversampling. Each test setting is tried 10 times.

Now I want to see if there is a statistical significance between the best oversampling model and no oversampling model (when looking at mean performance from 10 tests). Which test should I use to do this?

So basically I have these means from my tests and along with that I calculated the standard deviation from the results.

Test dataset is chosen at random, from two populations (710 and 520 000) each at 30%. Each ANN makes a binary classification and I get the AUC score. How can I see if the ANN's ability to predict this population, based on AUC, is statistically significant?

Improve this question
$\endgroup$
7
  • $\begingroup$ What do you need this test for? $\endgroup$
    – Tim ♦
    May 17, 2022 at 17:31
  • $\begingroup$ To see if oversampling is significant or not $\endgroup$
    – Don_twice
    May 17, 2022 at 18:54
  • $\begingroup$ What for? Why does it matter? $\endgroup$
    – Tim ♦
    May 18, 2022 at 10:11
  • $\begingroup$ Now I understand. I am completing a paper as part of my studies. I am allowed to get help and use it as long as I declare it as not mine, which I of course intend to do! Based on: ieeexplore.ieee.org/document/6790639/authors#authors it seems as if 5x2 cross validation along with a student t-test seems promising? However, I wonder if oversampling might effect this approximation? $\endgroup$
    – Don_twice
    May 18, 2022 at 12:21
  • $\begingroup$ Pardon me for rushing my answer, I read more than the abstract of my prior link. I think my experiment consists of comparing two classifiers and I have enough data for testing. As such I should perform a McNemar's test. However, the testing data needs to be the same for all classifiers tested. This could add errors since I am not using all data for testing. Perhaps perform k-fold cross validation and McNemar's test? Could this be something useful? $\endgroup$
    – Don_twice
    May 18, 2022 at 12:46

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.