I am a PhD student but I am having a hard time understanding certain aspects of statistics. I just ran an ANOVA and I got no significant interaction effect between Density (0 and 20) and Species (species A and species B).

However, Tukey's test shows that there is a significant difference between these variables. Species A at a density of 0 is significantly larger than the other combinations. Is this relevant given that ANOVA has already established that there is no significant interaction effect between Species and Density? Why would there be significant differences in Tukeys if ANOVA says there is no interaction effect in the first place?

  • 2
    $\begingroup$ Welcome to the site. I think you have a few things confused. Please show your code and a summary of the output. $\endgroup$ – Peter Flom Mar 16 '14 at 22:50
  • 1
    $\begingroup$ What code? I just need a general response. $\endgroup$ – user42040 Mar 16 '14 at 22:56
  • $\begingroup$ I want to see what difference you looked at with Tukey's $\endgroup$ – Peter Flom Mar 16 '14 at 23:03
  • $\begingroup$ I don't know how to copy and paste from JMP onto this. I am getting: Least Sq Mean Species A, Density 0 1.230 A Species B, Density 0 1.384 B Species A, Density 20 2.401 C Species B, Density 20 2.394 C Rows connected by the same letter are not significantly different. $\endgroup$ – user42040 Mar 16 '14 at 23:14
  • 8
    $\begingroup$ Just because you don't see the need for it doesn't mean that there's no reason to ask for it; there's several reasons why it might help @Peter formulate an answer. What you should do is put the information into your question, and if layout is important, indent the entire thing by 4 spaces (by clicking the $\{\}$ button with the entire section of output selected). $\endgroup$ – Glen_b Mar 16 '14 at 23:32

OK, just to confirm, I'm assuming the means you are comparing are:

(A,0) : 1.230 (B,0) : 1.384

(A,20): 2.401 (B,20): 2.394

The lines connecting the means are "practically" parallel (due to small sample size?), so no interaction effect between species( levels(A,B)) and density(levels(0,20)). But Tukey will probably show you differences between the means for (A,0) and (B,20), correct ? Or even between (A,0) and (B,0) ? Nothing unexpected.

The conclusions of no significant interaction could be two : 1) that instead of the full factorial model with interactions you need to reduce it to main effects model (do not use more complex model when it gives no gain, i.e. is "no different" from simpler model, or does not give any explanation of the sources of variability for the underlying process which generates the data) 2) you do not have enough data to say if there is interaction or not.

  • 3
    $\begingroup$ Out of curiosity, why two answers, rather than editing one answer? $\endgroup$ – Glen_b Mar 17 '14 at 4:01

It seems to me that you need to start with understanding the difference between main effects and interaction effects (actually, to meditate on the latter, or find someone who will explain the idea of interaction effects to you very well).

Interaction effect is present when changes in the response for different levels of one factor are not linearly moving along the other factor, i.e. higher density for species A gives effect of some size but higher density for species B gives effect of significantly different size, or even reverses the direction of the effect (e.g. mean response decreases instead of increasing), the two factors are then "interacting" with each other, or are not acting "independently" in the linear model.

The result you quote is possible. Is Tukey test showing siginifcant differences in the cells of the desing on the "extreme" (opposite) ends ? What does the interaction plot look like (no picture necessary, just quote the values of the means of the response variable in the cells of your design) ?

What values have you got in the ANOVA table and what are the results of Tukey's HSD?

  • 3
    $\begingroup$ And this illustrates why I asked for code; rather than needing to ask what the Tukey test is doing, I would know. Rather than asking about the interaction plot, I would know (or have a good guess). Nothing against @jgstat, just illustrating my point. $\endgroup$ – Peter Flom Mar 16 '14 at 23:38
  • 1
    $\begingroup$ +1 @PeterFlom precisely... (but it's @jgstat) $\endgroup$ – Glen_b Mar 17 '14 at 4:02

It isn't that the other answers are incorrect, per se, it is just that I think a more clear pedagogically oriented answer can be provided.

There are two commonly used Tukey's test in this type of situation. Tukey's HSD and Tukey's LSD. The LSD is less conservative than the HSD, similar to a full combinations of t-tests being performed in parallel. As you might imagine based on this description the LSD does not control for familywise error rate; the HSD does. So a lot hinges on which Tukey's test you are using. If you are using LSD, current tradition dictates you probably shouldn't be. If you are using HSD, we can keep going down this road.

In regards to your first question: "Is this relevant given that ANOVA has already established that there is no significant interaction effect between Species and Density?"...

Most 'ANOVA tradition' dictates that you not use either post-hoc unless a statistically significant effect has been found. An exception to this 'rule' is that tradition allows that if you had a good reason to compare groups at the outset before you looked at your data, you could simply do a t-test between those groups without controlling for familywise error rate. Also, if there was a good reason to compare one to many, there is a post hoc specifically for that approach (the name escapes me at the moment).

However, there is no denying that eyeballing your data you are seeing that the DV for Species A at a density of 0 is noticeably larger than the other combinations. So, you should mediate on this a little further. Do the DV means look different enough to you to be remarkable? What if you consider the degree of error in your measurements (e.g. (don't quote me I'm talking off the cuff) but, I think that sqrt MSE will give you an idea of what the ANOVA shrunken error)? Regardless, traditional approaches to ANOVA will not let you make any claims about this cell of the design versus the others or declare that they are significant; nevertheless, some fields are okay with you speculating based on non-significant effects. Your advisor, peers, and previous journal articles will give you an idea of how much rope you have in making your claims and observations.

For my part, I believe that now that you've looked at the data and are making decisions about how to proceed that you are on shaky ground in terms of having p-values that are directly meaningful. Ideally, you'd run the study again, this time with an a priori intent to check Species A at Density 0 against the other cells of your design. An alternative approach would be to use less familiar statistical methods that don't require you to plan your analyses ahead of time (e.g. Bayesian). But those beliefs of mine are a matter of opinion, so allow me to drift back to the facts.

Let's go on to your second question: "Why would there be significant differences in Tukeys if ANOVA says there is no interaction effect in the first place?"

This is one that even some seasoned scientists don't understand. Here is my shot at an explanation... the observed deviance between two cells that are maximally different relative to the grand mean will be on average higher than the observed deviance for four cells, three of which are very similar, relative to the grand mean. In short, the more cells you have that are similar in an interaction, the less likely you are to observe an statistically interaction even if a few cells /do/ reflect an interaction. I suspect your primary problem is that you used Tukey's LSD. Otherwise, the problem may be as jgstat suggested

This is unlikely to be your primary problem here because you have a pretty simple 2x2 design. I think your problem is probably that you used Tukey's LSD and so the differences between two cells happened to pop-out but that by-and-large the cells are mostly similar.

  • $\begingroup$ Wouldn't effect coding in multiple regression give you the one-vs.-many result of that elusive post hoc you were thinking of? The $t$-test of an effect code coefficient is of that group's mean vs. the grand mean (cf. UCLA Statistical Consulting Group). $\endgroup$ – Nick Stauner Mar 17 '14 at 4:22
  • 1
    $\begingroup$ Several approaches to do what I suggested is presented on pg 198: Type of Contrasts in Maxwell & Delaney, 2nd Edition (2004). The most familiar of these (to me at least) is the Scheffe post-hoc... but I have to admit that I haven't seriously thought about Scheffe in a /very/ long time and so I do not have the ability at the moment to evaluate M&D's claims regarding its utility in exactly this situation (e.g. when the desire to test a particular hypothesis is a result of having observed the data). $\endgroup$ – russellpierce Mar 18 '14 at 2:36
  • 1
    $\begingroup$ What I was actually thinking of was probably Dunnett's T3 (not D3). Hsu 1996 also looks tenable. I think it might be time for me to get a newer version of this book or better. $\endgroup$ – russellpierce Mar 18 '14 at 2:36
  • 1
    $\begingroup$ @NickStauner: Sorry Nick, I lost my response to you. In short I don't think that approach addresses the issue. First, the comparison of interest is not against the grand mean. Second, there is no familywise protection in standard regression and the ANOVA tradition tends to demand such protection. $\endgroup$ – russellpierce Mar 18 '14 at 2:37
  • $\begingroup$ I had a feeling my idea wasn't quite right (probably should've admitted this up-front). Good points; thanks! $\endgroup$ – Nick Stauner Mar 18 '14 at 6:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.