2
$\begingroup$

I have 4 models for a learning task, and I am performing pair-wise tests between them, on five metrics, to asses whether they present different distributions.

How should I go about multiple testing correction?

If I'm using the Bonferroni correction, should I use $m=5$ accounting for the five metrics? or should I use $m=4*5$ to account for all tests per model?

Improve this question
$\endgroup$
1
  • $\begingroup$ Both your answers seem incorrect. If you want to correct for overall error in the tests for each model, there are 10 pairwise comparisons so you should divide alpha by 10. If you want to correct for the whole family of 4*20=40 tests, divide alpha by 40. These are both conservative corrections. You need to decide what family if tests you want to control, and go from there. $\endgroup$
    – Russ Lenth
    Commented Nov 25, 2016 at 18:50

1 Answer 1

Reset to default
0
$\begingroup$

Partially answered in comments:

Both your answers seem incorrect. If you want to correct for overall error in the tests for each model, there are 10 pairwise comparisons so you should divide alpha by 10. If you want to correct for the whole family of 4*20=40 tests, divide alpha by 40. These are both conservative corrections. You need to decide what family if tests you want to control, and go from there. – rvl

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.