If I am comparing two groups with respect to 20 different variables, and do not adjust for the number of comparisons I make, will my probability of making a type-I error be larger than alpha = 0.05? Or will be equal or less than 0.05?


It will be larger, (if I understand you correctly saying that you will test each variable..)
You could view it as a 0.05 change to reject H0 unjustly for each comparison, this means 0.05n, with n being the number of comparisons.
You basically 'create' a possibility for a false positive with every comparison. This for example often happens with ANOVA (or when t-testing 100+ variables on an important outcome variable in hopes of finding some useful but unexpected predictor (common in datamining).
To remedy this you should use a form of a posteriori tests (post hoc tests).
E.g. Bonferroni's Procedure. What this does is effectively change your alpha (e.g. 0.05) to alpha / n (resulting for your example in 0.0025) As you can imagine the downside is loosing power..
Sidenote; this effect of increased type 1 error is sometimes called 'familywise' (or 'experimentwise') error rate

  • $\begingroup$ Thanks, Steven. That what I thought the correct answer was, that adjusting for 20 different variables, we will only reject the null hypothesis if p-value < 0.05/20 = 0.025. $\endgroup$ – saintacacia May 5 at 4:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.