If I am comparing two groups with respect to 20 different variables, and do not adjust for the number of comparisons I make, will my probability of making a type-I error be larger than alpha = 0.05? Or will be equal or less than 0.05?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
It will be larger, (if I understand you correctly saying that you will test each variable..)
You could view it as a 0.05 change to reject H0 unjustly for each comparison, this means 0.05n, with n being the number of comparisons.
You basically 'create' a possibility for a false positive with every comparison. This for example often happens with ANOVA (or when t-testing 100+ variables on an important outcome variable in hopes of finding some useful but unexpected predictor (common in datamining).
To remedy this you should use a form of a posteriori tests (post hoc tests).
E.g. Bonferroni's Procedure. What this does is effectively change your alpha (e.g. 0.05) to alpha / n (resulting for your example in 0.0025) As you can imagine the downside is loosing power..
Sidenote; this effect of increased type 1 error is sometimes called 'familywise' (or 'experimentwise') error rate
-
$\begingroup$ Thanks, Steven. That what I thought the correct answer was, that adjusting for 20 different variables, we will only reject the null hypothesis if p-value < 0.05/20 = 0.025. $\endgroup$ May 5, 2019 at 4:06