1
$\begingroup$

There are many questions and answers on how to apply p-value adjustments and in which cases, but I couldn't get an answer to my problem based on those, so here goes :

Assume you have a dataset with 5 quantitative variables and 2 qualitative variable (let's call them vquants and vquals for quick reference). You want to see if vqual-1 has an effect on any of your vquants, so you repeat something like an anova to check for each vquant. I understand this particular process alone could call for a p-value adjustment (such as Bonferroni for example).

Further in your analysis, you want to do the same thing for vqual-2.

Are the two sets of 5 anovas to be considered as different processes (and apply a $\alpha/5$ correction to both separately) or do you have to account for both sets as one big test repetition scenario (thus applying a $\alpha/10$ correction to all test)?

$\endgroup$
0
$\begingroup$

First question to answer here is 'Why we use corrections?'. We use them because a single test has non-zero probability of I-type error (rejecting null hypothesis while it shouldn't be, so called 'false discovery'). This is usually 5%.

If you make several tests you have (say) 5% probability of I-type error in each test. So this may lead to quite a large probability of making at least one I-type error. Applying correction allows you to keep probability of making at least one I-type error at about (say) 5%.

Now, moving back to your question, you have to decide if you want to have:

  • (say) 5% probability of at least one I-type error in all 10 tests (then $\alpha/10$ correction)

or

  • (say) 5% probability of at least one I-type error in 5 tests involving vqual-1 and another 5% probability of at least one I-type error in 5 tests involving vqual-2 (then $\alpha/5$ correction to both sets of tests)
$\endgroup$
  • $\begingroup$ So based on how I consider the contents of my analysis, I will have to choose different corrections ? If it's possible to do it either way, then is there anything else than just the type-I error's final correction to take into account for this decision ? $\endgroup$ – Romain B. Jun 7 '18 at 9:21
  • $\begingroup$ No, corrections only recuce type-I error, so this is the only issue to consider. Despite, type of correction (Bonferroni, Holm, ....), of course. $\endgroup$ – Łukasz Deryło Jun 7 '18 at 9:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.