3
$\begingroup$

I have been trying to compare the effect of a treatment over time, controlling for a continuous covariate. So two groups (treatment vs control) are given different treatments and compared pre and post. Then I add a continuous covariate into the equation to see if the original effect is still significant. I'm predicting an interaction effect whereby the control group is not different across time, but the treatment group is. The covariate is not of interest, but it is significantly different between groups so there's very likely a confound, so that's why i added it, but that's not why I'm here today.

Basically, the lme command from the nlme package is giving totally different F statistics to aov. I had thought that repeated measures anova was within the general linear model and so were the same as mixed models. My gut says to trust the results of the mixed model, but I don't know how to reconcile the difference. In my original data, it even goes from highly significant to non significant.

My groups are unbalanced but all people went through all time points. Also, my covariate is recorded individually at each time point. If any of those facts sets off any bells.

NOTE: this isn't an off topic question just about R, it's more about why ANOVA/ANCOVA is the same/different from mixed effects models. This probably has something to do with setting contrasts or specifying the model for the SS or something rather than anything to do with R.

Here is some mock data, that is 'like' my original data, but not exactly the same.

    id<-c(1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,12,12,13,13,14,14)
    group<-c(1,1,1,1,1,1,2,2,2,2,2,2,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2)##group 1 is control
    time<-c(1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2)
    outcome<-c(10,11,12,11,10,11,10,19,10,20,12,20,10,10,11,12,11,10,13,20,10,21,9,19,10,13,10,13)
    covariate<-c(80,82,80,83,79,80,70,80,69,75,69,80,80,84,80,81,80,82,79,85,72,85,72,85,72,73,73,74)

    mydata<-as.data.frame(cbind(id,group,time,outcome,covariate))
    summary(aov(outcome~factor(group)*factor(time)+covariate+Error(id)))
    anova(lme(outcome~factor(group)*factor(time)+covariate, random=~1|id, data=mydata))

Here is my output for ANOVA

    > summary(aov(outcome~factor(group)*factor(time)+covariate+Error(id)))

    Error: id
          Df Sum Sq Mean Sq
    factor(group)  1  4.255   4.255

    Error: Within
                       Df Sum Sq Mean Sq F value   Pr(>F)    
    factor(group)               1  94.98   94.98   57.99 1.33e-07 ***
    factor(time)                1 137.29  137.29   83.81 5.87e-09 ***
    covariate                   1  98.60   98.60   60.20 9.78e-08 ***
    factor(group):factor(time)  1  39.55   39.55   24.15 6.49e-05 ***
    Residuals                  22  36.04    1.64                     
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Here is my output for the mixed effects model

    > anova(lme(outcome~factor(group)*factor(time)+covariate, random=~1|id, data=mydata))
                       numDF denDF   F-value p-value
    (Intercept)                    1    12 1249.4127  <.0001
    factor(group)                  1    12   23.7548   4e-04
    factor(time)                   1    11  115.8036  <.0001
    covariate                      1    11   75.0274  <.0001
    factor(group):factor(time)     1    11   28.0040   3e-04

As you can see totally different. Any help would be greatly appreciated.

$\endgroup$
2
$\begingroup$

There are a few reasons as to why an ANCOVA might give you different results from an a linear mixed-effects model (LMM). ANOVAs/ANCOVAs require a very strict set of assumptions to be met for them to be appropriate for ones data, and these assumptions are rarely met, while LMMs are typically quite flexible. Due to these strict assumptions ANOVA/ANCOVAs aren't commonly used, at least not correctly.

I suspect that the main reason the two are giving very different results is that ANOVA's and ANCOVAs typically fall apart when the group sample sizes are different. Repeated measures ANOVA/ANCOVA can't cope with the fact that there are different numbers of individuals in each group within the time points measured. Mixed models can cope with this by altering the estimation for each Time × Group combinations variability.

This article goes through all the limitations of ANOVAs/ANCOVAs quite succinctly if you want to learn more about them. Let me know if you want any more clarification though!

TL;DR: ANOVAs can't cope with unbalanced groups, while LMMs can. Don't use ANOVAs/ANCOVAs.

$\endgroup$
  • 1
    $\begingroup$ Thanks for your comment, thumbs up, I figured out what it was: I had a time varying covariate. It was recorded on two separate occasions with different scores for each. When i just used the covariate from start time, it equaled. From a cursory glance of the internet, it appears that ANOVA can't handle time varying covariates at all, but if you know of anyway to get it to handle that, i would be interested. Thanks for the answer. $\endgroup$ – Huy Pham Feb 14 at 8:46
  • $\begingroup$ Glad you figured it out! What exactly is your 'outcome' measuring? Time-varying covariates are often used in survival analysis. If your outcome variable fits the requirements then perhaps this would be an avenue you could explore. $\endgroup$ – André.B Feb 17 at 20:33
  • $\begingroup$ Well it was a bunch of measures of athletic performance over time, the covariate was weight, but they measured weight twice. Makes sense; some people put on more muscle then others over time. Yeah i thought about survival, but then I'd have to chukn the continuous measures and the mixed model works fine, i just don't understand what summary(aov(outcome~factor(group)*factor(time)+covariate+Error(id))) is outputting then when the covariate changes. $\endgroup$ – Huy Pham Feb 18 at 5:20
  • $\begingroup$ You could just use the difference in weight between time point one and two. It would get around the issue of repeated measures because you could drop time as a variable and likely make the output more interpret-able. I would just stick with the mixed-model though - it is more appropriate given the design. $\endgroup$ – André.B Feb 18 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.