A two-way ANOVA yields 3 F ratios. Each F ratio has alpha of .05. Does that mean the experimentwise alpha level of the ANOVA is not .05 but rather is 1 - .95^3? If so, is there a recommended way to maintain the experimentwise alpha for two-way ANOVA?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
+1 to @PeterFlom for a good answer with several important pieces of information regarding the assumptions behind the question. Let me here address the question directly to complement that, bearing in mind that I fully agree with him.
One of the things I find frustrating is that the problem of multiple comparisons seems to be always discussed in terms of multiple pairwise comparisons after conducting a one-way ANOVA, which leads many people to believe that that is where the problem exists. This is just not true; the problem of multiple comparisons exists everywhere. For example, it applies just as much to factorial ANOVA as it does to multiple t-tests, and I appreciate the fact that you have intuited this. (That is, the answer to your initial underlying question is yes.)
The answer to your more specific question is that the experimentwise alpha is $1-(1-\alpha)^3$ only if the three tests are independent. (For example, the $n_{ij}$'s have to be equal.) If the tests are not independent, then the true experimentwise alpha is complicated to calculate, but cannot exceed $3\alpha$. Thus, for multiple independent contrasts, the Dunn-Sidak correction, $1-(1-\alpha)^{1/c}$ (where $c$ is the number of contrasts), is often recommended, but for multiple contrasts that are not all independent, the Bonferroni correction, $\alpha/c$, is recommended. The Dunn-sidak correction is more powerful, and using a step-down procedure is more powerful still, although the gains will be small when there are few contrasts to start with (such as $c=3$). I discuss these topics in more detail here.
There are many other approaches to the problem of multiple comparisons as well. There is also some debate regarding when the benefits outweigh the costs (i.e., lost power) as @PeterFlom mentions. If you had intended, a-priori, to test all three of your factors, and the $n_{ij}$'s are close to equal, I wouldn't worry much about this. Remember that your ANOVA comes with a global F-test, I think you should only feel uncomfortable moving forward if that test is 'non-significant'.
answered May 2, 2012 at 16:37
$\begingroup$
$\endgroup$
6
There are a whole bunch of methods for controlling alpha. The simplest is Bonferroni, but there are many others.
However, it's also questionable whether it needs to be controlled at all, and, if so, over what collection of tests.
I think p-values are almost always the answer to a question that we aren't interested in. Further, in most cases, the p-value is really not meaningful because the null isn't true.
answered Apr 2, 2012 at 10:23
-
$\begingroup$ In a 3 way ANOVA we have 3 main effects, 3 two-way interactions and 1 three-way interaction. That is 7 statistical tests, each done with alpha = .05. We would not do 7 t-tests, each with alpha = .05. Why do we blithely do 7 F tests each with alpha = .05? $\endgroup$– Joel W.Apr 5, 2012 at 19:11
-
$\begingroup$ 5 am thinking along the lines of Joel. In ANOVA the overall F test indicates whether or not there is any effect. If that is all you want to do there is no need to do multiplicity adjustment. But if the F test rejects the null hypothesis of no effect the investigator wants to know which variables are contributing to the effect. Statistical packages provide the option to test contrast in that situation. Multiplicity adjustment is necessary and the adjustment will tend to be larger as the number of contrasts tested is increased. $\endgroup$ May 5, 2012 at 15:23
-
$\begingroup$ In SAS the standard multiple comparison tests can be applied when using PROC ANOVA or PROC GLM. Also PROC MULTTEST will do p-value adjustment for FWER and FDR criteria using bootstrap, permutation, Bonferroni, Sidak and others. $\endgroup$ May 5, 2012 at 15:23
-
$\begingroup$ @JoelW. writes that we would not do 7 t-tests each with alpha = .05. In many fields we often DO just this, nor is it necessarily an error. The real error is using p-values to decide things, rather than effect sizes. $\endgroup$ May 5, 2012 at 22:00
-
$\begingroup$ @PeterFlom Certainly we need both. A large effect size that comes with a large p value is not something I would hang my hat on. Would you? $\endgroup$– Joel W.Dec 4, 2015 at 20:14