2
$\begingroup$

I have 4 models for a learning task, and I am performing pair-wise tests between them, on five metrics, to asses whether they present different distributions.

How should I go about multiple testing correction?

If I'm using the Bonferroni correction, should I use $m=5$ accounting for the five metrics? or should I use $m=4*5$ to account for all tests per model?

$\endgroup$
  • $\begingroup$ Both your answers seem incorrect. If you want to correct for overall error in the tests for each model, there are 10 pairwise comparisons so you should divide alpha by 10. If you want to correct for the whole family of 4*20=40 tests, divide alpha by 40. These are both conservative corrections. You need to decide what family if tests you want to control, and go from there. $\endgroup$ – rvl Nov 25 '16 at 18:50
0
$\begingroup$

Partially answered in comments:

Both your answers seem incorrect. If you want to correct for overall error in the tests for each model, there are 10 pairwise comparisons so you should divide alpha by 10. If you want to correct for the whole family of 4*20=40 tests, divide alpha by 40. These are both conservative corrections. You need to decide what family if tests you want to control, and go from there. – rvl

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.