# What qualifies as a set of independent tests to apply p-value adjustments?

There are many questions and answers on how to apply p-value adjustments and in which cases, but I couldn't get an answer to my problem based on those, so here goes :

Assume you have a dataset with 5 quantitative variables and 2 qualitative variable (let's call them vquants and vquals for quick reference). You want to see if vqual-1 has an effect on any of your vquants, so you repeat something like an anova to check for each vquant. I understand this particular process alone could call for a p-value adjustment (such as Bonferroni for example).

Further in your analysis, you want to do the same thing for vqual-2.

Are the two sets of 5 anovas to be considered as different processes (and apply a $\alpha/5$ correction to both separately) or do you have to account for both sets as one big test repetition scenario (thus applying a $\alpha/10$ correction to all test)?

First question to answer here is 'Why we use corrections?'. We use them because a single test has non-zero probability of I-type error (rejecting null hypothesis while it shouldn't be, so called 'false discovery'). This is usually 5%.

If you make several tests you have (say) 5% probability of I-type error in each test. So this may lead to quite a large probability of making at least one I-type error. Applying correction allows you to keep probability of making at least one I-type error at about (say) 5%.

Now, moving back to your question, you have to decide if you want to have:

• (say) 5% probability of at least one I-type error in all 10 tests (then $\alpha/10$ correction)

or

• (say) 5% probability of at least one I-type error in 5 tests involving vqual-1 and another 5% probability of at least one I-type error in 5 tests involving vqual-2 (then $\alpha/5$ correction to both sets of tests)
• So based on how I consider the contents of my analysis, I will have to choose different corrections ? If it's possible to do it either way, then is there anything else than just the type-I error's final correction to take into account for this decision ? – Romain B. Jun 7 '18 at 9:21
• No, corrections only recuce type-I error, so this is the only issue to consider. Despite, type of correction (Bonferroni, Holm, ....), of course. – Łukasz Deryło Jun 7 '18 at 9:26