1
$\begingroup$

I have different machine learning models and for each of them after a 10-fold cross validation I obtained a mean AUC (+-std). Now, How can I check if there is a statistically significant difference (p-value) among the mean AUC?

$\endgroup$
4
  • $\begingroup$ I wouldn't do this. Anyway, can you post some sample data? $\endgroup$ Apr 3, 2020 at 10:52
  • $\begingroup$ Example of AUC for three models: 0.66 (±0.04) 0.74 (±0.04) 0.77 (±0.04) $\endgroup$
    – zorro
    Apr 3, 2020 at 11:11
  • $\begingroup$ Do you only have these statistics or can you also obtain the raw 10 values for each fold? $\endgroup$ Apr 3, 2020 at 11:18
  • $\begingroup$ Unfortunately I saved only the mean value. I could get the individual measurements on each test set but I have to re-calculate them for each model. $\endgroup$
    – zorro
    Apr 3, 2020 at 11:29

1 Answer 1

1
$\begingroup$

Don't.

A hypothesis test aims to test an hypothesis, or rather the null hypothesis (this is beside the point here). But you know that your models are different because, as you write, you have different machine-learning models.

I guess what you want to know is something like whether approach A is consistently, or on average, better than B. That is what the numbers (mean and se) tell you. More importantly, they also tell you how much better A is than B. You don't get any further information from knowing the hypothesis-test results.

Side note: As AUC approaches 1, estimates become non-normally distributed, making the sem less useful than the confidence interval. Also, sensitivity to differences decreases as AUC approaches 1, and I have seen people compare $e^\text{AUC}$ rather than AUC itself in this situation.

$\endgroup$
1
  • $\begingroup$ I have to confront different models considering a metric (AUC or sensitivity etc) on the same data, but also on different data. I should use a hypothesis test to determine if my models are significantly different from one another. An AUC of 0.8 is significantly different from 0.75 ? To assess whether the difference in AUC between highest performing classifier and the other methods was statistically significant, someone uses the Wilcoxon signed-rank test. Other DeLong method. I am really confusing. $\endgroup$
    – zorro
    Apr 3, 2020 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.