7
$\begingroup$

Study Design: I have a 2x3 factorial design, 2 levels of Time (2050 or 2100) by 3 levels of information (None/Control, Moderate, Extreme).

I set up some very specific contrasts when analyzing this design, in particular Control vs. Other, Moderate vs. Extreme, and 2050 vs. 2100. For one DV, the overall ANOVA was not significant but the Control vs. Other contrast was.

Question: What is the best way to interpret a nonsignificant ANOVA but a significant contrast?

I know that ANOVA is used to lower familywise error rates, and I wouldn't want to fall into the trap of ignoring the overall ANOVA for pairwise comparisons. However, this is one of a select few contrasts that I planned ahead of time. Does that make a difference in interpretation, or should I simply consider the manipulation not significant?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ There are a lot of posts on this site that ask, essentially, this same question. Here is one example: possible duplicate of How can I get a significant overall ANOVA but no significant pairwise differences with Tukey's procedure? $\endgroup$
    – Macro
    Jun 8, 2012 at 1:59
  • 3
    $\begingroup$ Good catch, Macro. The present question, however, appears to focus on a different aspect of this issue: interpretation. Given that the replies to the duplicate do not appear to address this at all, I am reluctant to vote to close. $\endgroup$
    – whuber ♦
    Jun 8, 2012 at 12:51
  • 1
    $\begingroup$ Also, the question you suggest as a duplicate is something of an "opposite" -- it asks what to do when ANOVA is significant but post-hocs are not. In this case I was asking how to interpret it when the ANOVA is non-significant but the post-hocs are. $\endgroup$
    – Lee
    Jun 15, 2012 at 20:59

2 Answers 2

Reset to default
9
$\begingroup$

The answer to your question reflects your view of overall (or experimentwise) alpha. If, before you collected your data, you set forth planned comparisons (and it sounds like you did that), then after you collect the data you do those and only those comparisons, and there is no reason to look at any other comparisons (AKA contrasts) nor at the overall ANOVA. Why did you look at the overall ANOVA? If you did both planned comparisons and ad hoc comparisons, then you inflated your experimentwise alpha, which some consider a serious error. Of course, you might do ad hoc comparisons to help you plan your future research, but not report them. Also, it seems you have two DVs and did two ANOVAs. In general, if you do two ANOVAs each with p = .05 then you have inflated your experimentwise alpha. Therefore, when doing 2 ANOVAs you might reduce the alpha level for each of your two ANOVAs to maintain your experimentwise alpha. Alternatively, you might do a MANOVA (i.e., a multivariate analysis of variance) which allows you to consider both DVs simultaneously. Of course, you should also be looking at effect sizes, to help you understand the meaning of your data.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Perhaps I am mistaken, but I was under the impression that one should always look at the overall ANOVA and then interpret contrasts/post-hoc comparisons. But then again, I generally do post-hoc comparisons only. So it's good to know that contrasts alone are the only comparisons I should look at. Do these retain the same alpha level (.05) or should they be reduced? Also, thanks, I am well aware of the multiple ANOVA problem and the possibility of MANOVA. $\endgroup$
    – Lee
    May 21, 2012 at 23:16
  • 1
    $\begingroup$ Yes. You do need to control the experimentwise alpha when doing planned comparisons. $\endgroup$
    – Joel W.
    May 29, 2012 at 10:23
  • 5
    $\begingroup$ The key nuance here, is whether you have "planned comparisons" (as JoelW. discusses), or "post-hoc comparisons" (like you mention). If your comparisons were decided a-priori (& they aren't just to look at all pairwise contrasts), then JoelW.'s advice is good, but if you gathered some data, looked at it, noticed that some contrasts looked interesting & decided to test those, then you need to exercise more caution: eg, no post-hoc tests w/ non-significant ANOVA, & use alpha adjustments, etc. $\endgroup$
    – gung - Reinstate Monica
    Jun 8, 2012 at 1:47
  • $\begingroup$ Thank you for the answer to Lee's question, which parallels mine. Do you happen to know of any published work that would justify looking at planned comparisons only, even in the absence of a significant three-way interaction in ANOVA? Thank you in advance for whatever recommendations. $\endgroup$
    – user30056
    Sep 8, 2013 at 17:20
0
$\begingroup$

Yes you need to adjust alpha, especially if you do planned comparisons that are non-orthogonal. If you have set the contrasts in an ANOVA and tested the orthogonality between them, you can descompose the SC and df of the ANOVA. The planned comparison are always more powerful than traditional post hoc test, but in cases where the contrast are non.orthogonal you have inflated the experimentwise error. you can check for orthogonality among contrast in many textboxs. However i don´t undestand if you have set that time was factor with two levels, why are you performing a contrast between those times?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.