Suppose I do $1000$ normality tests, and set $\alpha = 0.05$. When testing for normality, should I correct for multiple comparisons? So if $p>\frac{0.05}{1000}$, I should keep the null hypothesis (think the data is normally distributed)? I am using the Shapiro-Wilks test.
4
-
2$\begingroup$ Why are you performing these normality tests? $\endgroup$– EdMCommented Jun 21, 2015 at 19:11
-
1$\begingroup$ Whether one should use multiple comparisons when doing multiple tests depends on the purpose you're using it for. What are you testing for normality, and (as EdM also asks), why are you testing that? $\endgroup$– Glen_bCommented Jun 21, 2015 at 23:36
-
$\begingroup$ I need to decide between parametric and non-parametric test for 1,000 variables. I know that normality tests do not actually answer should I pick parameteric or non-parametric test but it is best option I have (given the too large data to be manually checked). I think most variables are normally distributed. I would like to find the few non-normal ones, without a huge number of false positives. $\endgroup$– norCommented Jun 22, 2015 at 7:14
-
$\begingroup$ @nor check this thread, you may find it interesting: stats.stackexchange.com/questions/2492/… $\endgroup$– TimCommented Jun 22, 2015 at 18:15
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The answer depends on what risks you are trying to minimize. In a typical situation with multiple testing you are trying to avoid false-positive findings, so you take into account the fact that tests at $\alpha = 0.05$ might give a false-positive result 5% of the time.
In your case, however, you want to identify variables that are non-normally distributed so that you will use non-parametric tests on those variables. If you think that identifying non-normal distributions is very important, it seems that you would want to minimize false-negative findings (that is, you don't want to miss variables that really are non-normal). In that case adjusting for multiple comparisons works against that goal. One might even argue that $\alpha = 0.05$ is not sufficiently stringent, if you believe that applying a parametric test to a variable without a normal distribution is really risky.
That said, you might be worrying too much about the normal distributions of the variables, depending on the specifics of your analysis. Some argue that testing for normal distributions of variables is often useless; you should read that question and its answers before you proceed. In linear regression, normal distributions of variables themselves are not required, although normal distributions of residual errors are needed for some types of significance tests. If some of your variables are categorical, they can't be normally distributed in any event. There's a nice example here of useful linear regression on variables that aren't normally distributed. The t-test can be robust against lack of normality, depending on the variable distributions. So evaluate carefully whether you should even be doing this testing.
