8
$\begingroup$

What (with justification) is a valid post-hoc for a two-way ANOVA main effect, if no interaction is present?

Example two-way fixed effect ANOVA:

> aov.example <- aov(Response ~ IV1 * IV2, data=data)
> summary(aov.example)

              Df Sum Sq Mean Sq F value   Pr(>F)   
IV1            1  13.10  13.099  0.7222 0.40547  
IV2            4 315.56  78.891  4.3498 0.01081 *
IV1:IV2        4 141.00  35.251  1.9436 0.14240  
Residuals     20 362.74  18.137          
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

I see a few general options, but I'm unsure which is most appropriate:

  1. Post-hoc with a multiple comparison test using two-way model

    > TukeyHSD(aov.example, which="IV2")

Tukey multiple comparisons of means
   95% family-wise confidence level

Fit: aov(formula = Response ~ IV1 * IV2, data = data)

$IV2
               diff        lwr        upr     p adj
B-A      -2.1485711  -9.506162  5.2090200 0.9031415
C-A      -2.3382727  -9.695864  5.0193184 0.8733517
D-A       1.4732257  -5.884365  8.8308168 0.9735981
E-A      -8.0515205 -15.409112 -0.6939294 0.0277241
C-B      -0.1897016  -7.547293  7.1678895 0.9999912
D-B       3.6217968  -3.735794 10.9793879 0.5905067
E-B      -5.9029494 -13.260541  1.4546416 0.1559460
D-C       3.8114984  -3.546093 11.1690895 0.5439632
E-C      -5.7132478 -13.070839  1.6443432 0.1785266
E-D      -9.5247462 -16.882337 -2.1671552 0.0074740

  2. Post-hoc with a multiple comparison test using one-way model

    > TukeyHSD(aov(Response ~ IV2, data=data))

Tukey multiple comparisons of means
   95% family-wise confidence level

Fit: aov(formula = Response ~ IV2, data = data)

$IV2
               diff        lwr        upr     p adj
B-A      -2.1485711  -9.858179  5.5610365 0.9224401
C-A      -2.3382727 -10.047880  5.3713349 0.8976378
D-A       1.4732257  -6.236382  9.1828333 0.9794638
E-A      -8.0515205 -15.761128 -0.3419130 0.0375667
C-B      -0.1897016  -7.899309  7.5199060 0.9999933
D-B       3.6217968  -4.087811 11.3314044 0.6456949
E-B      -5.9029494 -13.612557  1.8066581 0.1951992
D-C       3.8114984  -3.898109 11.5211060 0.6014671
E-C      -5.7132478 -13.422855  1.9963597 0.2212053
E-D      -9.5247462 -17.234354 -1.8151387 0.0102178

    Overall, in this example the significant comparisons do not change, but the adjusted p-values are lower using the two-way ANOVA model. Which is most appropriate for a two-way main effect post-hoc?

  3. Post-hoc with subset levels of other independent variable

I'm pretty sure this is not valid, given the lack of any effect across the other independent variable. However, when one level of IV1 has larger effects between levels in IV2 compared to other IV1 levels, wouldn't this strongly skew the overall analysis? In other words, even though overall there is a main effect, could this effect not be significant in the the other level of IV1?

    > aov.level1 <- aov(Response ~ IV2, data=data[1:15,])
    > summary(aov.level1)

                Df Sum Sq Mean Sq F value  Pr(>F)  
    IV2          4 334.55  83.639  3.8413 0.03837 *
    Residuals   10 217.74  21.774                  

    > aov.level2 <- aov(Response ~ IV2, data=data[16:30,])
    > summary(aov.level2)

                Df Sum Sq Mean Sq F value Pr(>F)  
    IV2          4 122.01  30.503  2.1037 0.1551
    Residuals   10 145.00  14.500

Level 1 of IV1 has a significant IV2 main effect, but Level 2 does not. There are also differences in significant multiple comparisons. My concern is that I'm committing a Type I error using the previous two methods.

    > TukeyHSD(aov.level1)

    Tukey multiple comparisons of means
        95% family-wise confidence level

    Fit: aov(formula = Response ~ IV2, data = data[1:15, ])

    $IV2
                     diff        lwr         upr     p adj
    B-A       -7.86237771 -20.401217  4.67646157 0.3054028
    C-A       -7.90634218 -20.445181  4.63249709 0.3008150
    D-A       -0.94962690 -13.488466 11.58921237 0.9989922
    E-A      -12.55848654 -25.097326 -0.01964727 0.0496014
    C-B       -0.04396448 -12.582804 12.49487479 1.0000000
    D-B        6.91275080  -5.626088 19.45159008 0.4167559
    E-B       -4.69610883 -17.234948  7.84273044 0.7342625
    D-C        6.95671528  -5.582124 19.49555455 0.4111062
    E-C       -4.65214436 -17.190984  7.88669492 0.7404793
    E-D      -11.60885964 -24.147699  0.92997964 0.0729541

    > TukeyHSD(aov.level2)

    Tukey multiple comparisons of means
        95% family-wise confidence level

    Fit: aov(formula = Response ~ IV2, data = data[16:30, ])

    $IV2
                   diff        lwr       upr     p adj
    B-A       3.5652355  -6.667185 13.797656 0.7795224
    C-A       3.2297968  -7.002624 13.462218 0.8321507
    D-A       3.8960783  -6.336343 14.128499 0.7231230
    E-A      -3.5445545 -13.776975  6.687866 0.7829174
    C-B      -0.3354387 -10.567860  9.896982 0.9999634
    D-B       0.3308428  -9.901578 10.563264 0.9999654
    E-B      -7.1097900 -17.342211  3.122631 0.2257310
    D-C       0.6662815  -9.566139 10.898702 0.9994435
    E-C      -6.7743513 -17.006772  3.458070 0.2618999
    E-D      -7.4406328 -17.673054  2.791788 0.1942037

This post has some relevant information.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ +1, for a question that I think is important, and clearly & fully stated. One thing I will say right off is that, re #3, if you are finding differences in the pattern of results across the two different levels of IV1, you probably have a type II error in finding the interaction 'non-significant'. Either the pattern is the same at both levels of IV1, or the interaction obtains. $\endgroup$
    – gung - Reinstate Monica
    Commented May 17, 2012 at 14:57
  • $\begingroup$ No, because the interaction tests whether the differences in the pattern are different, which is what you want to know. Simply finding significant effects in one case and not in another (even if direction is changed) is not sufficient to test whether the two cases are different. The interaction does address that. Although, depending on how one interprets Type II error I'm not arguing there's none here... since they almost always exist when we don't reject null. $\endgroup$
    – John
    Commented May 17, 2012 at 20:16

2 Answers 2

Reset to default
2
$\begingroup$

You have no interaction and no main effect if IV1. Do NOT go looking at all comparisons of all means.

Even if you stick with IV2 I'm concerned that you're comparing significant and not significant and drawing conclusions and the difference between significant and not significant is likely not significant. Look at your first set of comparisons there. Do you want to make some conclusion about E-A being different from E-B? You can't from what you've presented. That's then another layer of testing. Have you even considered that it just might be E-ABCD is the main thing here? There's another layer of testing. You need to use some theory and narrow down what you're looking for. Or, you need a completely different approach.

The first thing you should do is step back and look at the pattern of data. The significant main effect does NOT mean that there are two means different in there somewhere. And even if there were, it doesn't mean that's what the main effect is. It only means that the pattern of results observed is unlikely to have occurred if there really were no differences. You're fishing around for individual comparisons and haven't even described the data yet. How about just describing the pattern you observed and saying it was unlikely to have occurred by chance? That's reporting your main effect. You have to have a reason for going and rooting around for individual effects.

What you're doing actually defeats the whole purpose of ANOVA. Consider that the ANOVA is run because doing multiple tests is subject to multiple comparison issues. So, you do the ANOVA, find an effect, and then go ahead and do all 1-1 multiple tests. Stop. The ANOVA tells you something. It tells you the pattern of data mean something. If you have a dram of theory and describe the pattern of data you might just avoid all of these tests.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the detailed answer, this is pretty much what I'm looking for! As I mentioned in a previous comment, the basic research question in this example: "Is there a difference between any of the treatments (IV2) within level 1 or 2 of IV1? Do these differences vary between level 1 and 2?" A two-way ANOVA seems to answer the general question, but determining where the differences are is important. For the post-hoc of the main effect, given the research question, would you suggest using the two-way or the one-way model? $\endgroup$
    – RobJackson28
    Commented May 17, 2012 at 19:41
  • 1
    $\begingroup$ I think you should have run an experiment with more power if any of what you have are really meaningful effect sizes. :) That said, for ANOVA such things almost never matter much and it's clear in your findings that's the case. Given your question, and the likelihood that you should be presenting all condition means with confidence itnervals, I'd go with the full model anyway. But there are certainly arguments that can be made to the contrary. If you were genuinely concerned about presenting what you feel is the most parsimonious model I'd go with the one way. $\endgroup$
    – John
    Commented May 17, 2012 at 20:27
0
$\begingroup$

I don't like the term appropriate post hoc analysis. Generally speaking if you tested and found a statistically significant main effect but no significant first order interactions you might be interested in some pairwise difference to identify why there is an effect.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I share the sentiment and have edited the question for clarity, but to be pragmatic how would you typically proceed with the pairwise comparisons (or planned to have proceeded)? Two-way model, one-way model, subset for each level of the other independent variable, or something else entirely? I understand that these decisions need to be made a priori, but I want to be able to justify that decision before conducting an experiment (rather than proceed with any conventional option without understanding why the choice is made). $\endgroup$
    – RobJackson28
    Commented May 17, 2012 at 13:28
  • $\begingroup$ I don't think that is a question to be asking us. Which tests to use depends on the problem. What is the investigator interested in knowing ? Is there specific hypotheses that he wants to test to make for an interesting paper? If an ANOVA is done and is appropriate we know that a difference between group means was a question to be answered. Followup analyses should be decided by the investigator and not some far removed statistician. $\endgroup$
    – Michael R. Chernick
    Commented May 17, 2012 at 15:58
  • $\begingroup$ I completely agree that it needs to be decided by the investigator; but as a prospective investigator, how can I decide which follow-up analyses can best answer my research question? For example, in this example experiment, the general question is: Is there a difference between any of the treatments within level 1 or 2? Do these differences vary between level 1 and 2? Overall, since it excludes particular post-hoc testing, what is the use of a global test (other than convention) if the multiple comparisons desired are decided a priori? $\endgroup$
    – RobJackson28
    Commented May 17, 2012 at 16:41
  • $\begingroup$ Relevant question $\endgroup$
    – RobJackson28
    Commented May 17, 2012 at 16:41
  • $\begingroup$ I think that to answer this new question I would say that the global test is more than just convention. If you are looking for differences between three or more groups, expect differences but are not sure which groups differ it makes sense to do the global test first, because if you can't find an overall difference there is no point search through the pairwise differences. If there is rejection of the global null hypothesis then you have a good reason to explore to try to identify where the difference lies. $\endgroup$
    – Michael R. Chernick
    Commented May 17, 2012 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.