0
$\begingroup$

I’m comparing the performance of 10 ML models across 15-fold cross-validation, using metrics like MSE. Each model’s performance is ranked per fold, and I want to determine if there are significant differences in performance.

I’m considering:

Friedman test for overall differences between models followed by:

  1. Conover
  2. Nemenyi
  3. Wilcoxon signed-rank test with multiple testing correction (e.g. Bonferroni)
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

My recommendation would be to start with the Friedman test to establish whether there are any overall differences in the ranks of the models across the folds since this will account for 15 "repeated measures." Next, move on to post-hoc analyses with the choice of test depending on your goals:

  1. Conover is good if your goal is for higher sensitivity and are okay applying a multiple-testing correction.
  2. Nemenyi is fine if you would rather have some nice visualizations that are produced in the critical difference diagrams.
  3. Wilcoxon for pairwise testing is appropriate if you need very precise pairwise comparisons and are again okay adjusting for multiple-testing.

In the first and third cases, apply the Holm correction or Bonferroni correction. The Bonferroni adjustment is more conservative than the Holm correction, and the Holm procedure is more powerful. See here for a discussion on CV of the two approaches.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for your answer! Why can I do the critical difference diagram only with Nemenyi? All I need is there pairwise p values? Is dependence an issue for any of these tests? $\endgroup$
    – PascalIv
    Commented Nov 26 at 8:17
  • $\begingroup$ You can do certainly create critical difference diagrams with other methods, no problem. I should have been more clear -- I usually create those particular graphics with the Nemenyi test which is why I mentioned it. To avoid problems with dependence, you'll want to aggregate the CV metrics for each model (e.g., mean accuracy across folds) and then compare the aggregates using the post-hoc tests. Each aggregate value is independent of the other since each represents an overall summary of performance. You could also try a bootstrap test or mixed effects model to account for dependence. $\endgroup$
    – StatsStudent
    Commented Nov 26 at 18:55
  • $\begingroup$ You could also consider using a Bayesian approach or hierarchical model to naturally handle the dependencies. Alternatively, you could use nested cross-validation to ensure independence. $\endgroup$
    – StatsStudent
    Commented Nov 26 at 18:58
  • $\begingroup$ Thanks again! Maybe I misunderstand that, I can not aggregate the cv metrics because that is the variation I want to test! I am not sure how nested cross validation changes anything. Still the training datasets will have an overlap leading to dependence $\endgroup$
    – PascalIv
    Commented Nov 27 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.