3
$\begingroup$

I want to use both anova and linear models to test the assumption that my some of my categories have different means than the rest.

I am using stats::aov for anova and nlme::lme for linear modelling. The full code is available in this notebook.

Basically I end up getting:

Error: ID
          Df  Sum Sq  Mean Sq F value Pr(>F)
Residuals  6 0.01198 0.001997               

Error: Within
          Df  Sum Sq  Mean Sq F value Pr(>F)  
COI        7 0.01926 0.002752    1.92 0.0903 .
Residuals 42 0.06020 0.001433 

and

Fixed effects: ER ~ COI 
                      Value  Std.Error DF   t-value p-value
(Intercept)      0.00200000 0.01465712 42 0.1364525  0.8921
COIemotion-hard  0.05600000 0.02023699 42 2.7672098  0.0084
COIscrambling-06 0.04218045 0.02023699 42 2.0843242  0.0433
COIscrambling-10 0.02094737 0.02023699 42 1.0351029  0.3065
COIscrambling-14 0.00685714 0.02023699 42 0.3388420  0.7364
COIscrambling-18 0.02085714 0.02023699 42 1.0306445  0.3086
COIscrambling-22 0.04885714 0.02023699 42 2.4142493  0.0202
COIscrambling-26 0.02742857 0.02023699 42 1.3553681  0.1825

Which means that lme is telling me 3 groups may significantly lie outside the range of the others, while aov is telling me the probability of ANY GROUP AT ALL being different from the others is not significant (using a p=0.05 cutoff).

What am I to make of this? Do I understand these results correctly?

$\endgroup$
  • 1
    $\begingroup$ That's probably because you're testing the combined effect (7 degrees of freedom) at .05 in the first case, whereas you're testing each difference with the reference category separately at .05, 1 degree of freedom at a time (more power, but also higher Type I error rate) $\endgroup$ – Patrick Coulombe Nov 17 '13 at 18:49
  • 3
    $\begingroup$ It is a well-known fact that a non-significant omnibus test does NOT imply that none of the comparisons subsumed under that omnibus test are significant. This has nothing to do with aov() vs. lme(). $\endgroup$ – Jake Westfall Nov 17 '13 at 18:50
  • $\begingroup$ Is CDIscrambling actually a continuous variable that you've made categorical? To be clear, it's still continuous if, even though you've collected values 06, 10, 14, etc., other values are possible. $\endgroup$ – John Nov 17 '13 at 19:28
  • $\begingroup$ so what should I trust? lme or aov? Are the inflated lme significances because of multiple comparison bias? is lme making the same error I would make by doing pairwise t-tests? $\endgroup$ – TheChymera Nov 17 '13 at 19:57
  • $\begingroup$ You don't have to choose because the two methods do not contradict each other in any way. One presents the omnibus test while the other tests the individual contrasts. As I already said: "It is a well-known fact that a non-significant omnibus test does NOT imply that none of the comparisons subsumed under that omnibus test are significant." The following document I found online may be useful to you: www4.uwsp.edu/psych/cw/statmanual/… $\endgroup$ – Jake Westfall Nov 17 '13 at 22:45
2
$\begingroup$

The analyses being presented aren't being fairly compared so there's no way to know if they're actually saying different things. They could be perfectly compatible. You could use lm (what underlies aov) and get the treatment effects just as you have for lme. You can also find omnibus F's using lme. Then you could compare things a little better but that's not really how you decide between mixed models and repeated measures ANOVA for a particular case. Regardless, the omnibus F in an ANOVA is not about simple effects so what you're comparing is apples and pizza.

The lme and aov commands use very different kinds of analysis methods. Baayen (2008) and others have demonstrated that the mixed models used in lme are generally more powerful compared to conventional repeated measures ANOVA and they also have fewer assumptions (e.g. sphericity). That said, there is the issue that mixed models are more sensitive to normality in residuals partially because with repeated measures ANOVA you benefit from the CLT to generate normality. There are many differences and the modelling methods function differently. You should be selecting one based on theoretical grounds and not particular results. A few years ago I was pretty much told by every statistician I met that you just should never use repeated measures ANOVA. I don't think that's as universal advice as they thought it was but perhaps that will help you.

It is possible that lme is finding effects not found in your simple ANOVA using aov. It's not practically possible to get truly comparable results to decide what to believe because they are doing different things and any tests mean (slightly) different things. That said, you could at least try anova(mylmeModel) to get omnibus F tests of the mixed effects model. Then you've got something substantially more comparable than what you're presenting here.

NOTE: I also asked if CDIscrambling was a continuous variable because it's a terrible habit of those who typically use repeated measures ANOVA to turn continuous variables into multiple categories. When using mixed models don't do that. You'll get much more power out of being able to treat it as a continuous variable. If it's not linear that's a different question and you can deal with that separately.

$\endgroup$
  • $\begingroup$ The continuousness is a bit complicated, emotion is continuous and scrambling is continuous. And I plan to do a ER~scrambling*emotion later. But right now I just want to see if any one category is special and cannot be compared to the others by these continuous metrics because at 100% emotion something different (qualitatively) happens than at 40% emotion and all scrambling steps. $\endgroup$ – TheChymera Nov 18 '13 at 10:54
  • $\begingroup$ Perhaps somethings special happens but if that's your question you should really make a new one about that. Nothing you've presented here can come close to answering that. In the new question try to define what makes one "special" and add a plot or plots. Also mention why your categorization is theoretically important for the specialness. Aside, you seem to be taking the effects in the regression as the effects and imbuing them with extra meaning. They're just each level compared to the first in the sorted factor. That's unlikely a good way to define special. $\endgroup$ – John Nov 18 '13 at 12:22
  • $\begingroup$ I actually already did ask the question, though I did fail to include a figure (tell me if you still think the question needs one after reading it) and that is where I got the suggestion of using a linear model instead of anova. You seem to hold that linear models aren't the right way of going at this - what is? $\endgroup$ – TheChymera Nov 18 '13 at 14:15
  • $\begingroup$ And here is actually an other very similar question of mine with a figure reflecting what my data looks like. I'm actually sorry for asking similar questions over and over again, but with every new method I try it turns out that's not right either :-/ $\endgroup$ – TheChymera Nov 18 '13 at 14:18
  • $\begingroup$ Also, by your initial answer do you mean to say that linear models are not an alternative to anova but rather a sub-calss which anova uses - and that lme4/nlme ARE in essence ANOVAS but they simply use a linear model which is superior to aov's? could I get the same kid of output nlme::lme gives me from stats::aov $\endgroup$ – TheChymera Nov 18 '13 at 14:44
2
$\begingroup$

(Converted from a set of comments.)

You don't have to choose because the two methods do not contradict each other in any way. One presents the omnibus test while the other tests the individual contrasts. It is a well-known fact that a non-significant omnibus test does NOT imply that none of the comparisons subsumed under that omnibus test are significant. This has nothing to do with aov() vs. lme().

The following document I found online may be useful to you:

The Meaning of the Omnibus F

wouldn't what you are saying amount to "it is normal for omnibus tests (here ANOVA) to lead to false negatives"?

Yes.

so if I run lme I would have to define a priori which conditions interest me, and simply picking the most significant ones a posteriori means I'm data mining?

Yes.

$\endgroup$
  • $\begingroup$ All hypothesis tests tend to find a preponderance of false negatives. There's very little effort put out to control the beta rate. $\endgroup$ – John Nov 18 '13 at 3:08
  • $\begingroup$ @John Hear hear! $\endgroup$ – Jake Westfall Nov 18 '13 at 3:13
  • $\begingroup$ Just to be clear: I choose the "COI" as the measure of interest (a priori). In the plot I see that one category (A) in "COI" is always null. To test whether it truly is outside the range of all the others I run aov. It kind of isn't (p=0.09). I have heard linear models are better than anova, so I do lme. I find out that lme says the most likely categories to be out of range are B, C, and G - and in fact, not A. What conclusion can I draw from this? Also here 3 of 8 categories are out of range .Can 4 of 8 be out of range? If so which ones will lme anotate as significant? $\endgroup$ – TheChymera Nov 18 '13 at 11:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.