# Models comparison and multiple testing correction

I have 4 models for a learning task, and I am performing pair-wise tests between them, on five metrics, to asses whether they present different distributions.

How should I go about multiple testing correction?

If I'm using the Bonferroni correction, should I use $m=5$ accounting for the five metrics? or should I use $m=4*5$ to account for all tests per model?

• Both your answers seem incorrect. If you want to correct for overall error in the tests for each model, there are 10 pairwise comparisons so you should divide alpha by 10. If you want to correct for the whole family of 4*20=40 tests, divide alpha by 40. These are both conservative corrections. You need to decide what family if tests you want to control, and go from there. – rvl Nov 25 '16 at 18:50

## 1 Answer

Partially answered in comments:

Both your answers seem incorrect. If you want to correct for overall error in the tests for each model, there are 10 pairwise comparisons so you should divide alpha by 10. If you want to correct for the whole family of 4*20=40 tests, divide alpha by 40. These are both conservative corrections. You need to decide what family if tests you want to control, and go from there. – rvl