2
$\begingroup$

I have been trying to compare the effect of a treatment over time, controlling for a continuous covariate. So two groups (treatment vs control) are given different treatments and compared pre and post. Then I add a continuous covariate into the equation to see if the original effect is still significant. I'm predicting an interaction effect whereby the control group is not different across time, but the treatment group is. The covariate is not of interest, but it is significantly different between groups so there's very likely a confound, so that's why i added it, but that's not why I'm here today.

Basically, the lme command from the nlme package is giving totally different F statistics to aov. I had thought that repeated measures anova was within the general linear model and so were the same as mixed models. My gut says to trust the results of the mixed model, but I don't know how to reconcile the difference. In my original data, it even goes from highly significant to non significant.

My groups are unbalanced but all people went through all time points. Also, my covariate is recorded individually at each time point. If any of those facts sets off any bells.

NOTE: this isn't an off topic question just about R, it's more about why ANOVA/ANCOVA is the same/different from mixed effects models. This probably has something to do with setting contrasts or specifying the model for the SS or something rather than anything to do with R.

Here is some mock data, that is 'like' my original data, but not exactly the same.

    id<-c(1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,12,12,13,13,14,14)
    group<-c(1,1,1,1,1,1,2,2,2,2,2,2,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2)##group 1 is control
    time<-c(1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2)
    outcome<-c(10,11,12,11,10,11,10,19,10,20,12,20,10,10,11,12,11,10,13,20,10,21,9,19,10,13,10,13)
    covariate<-c(80,82,80,83,79,80,70,80,69,75,69,80,80,84,80,81,80,82,79,85,72,85,72,85,72,73,73,74)

    mydata<-as.data.frame(cbind(id,group,time,outcome,covariate))
    summary(aov(outcome~factor(group)*factor(time)+covariate+Error(id)))
    anova(lme(outcome~factor(group)*factor(time)+covariate, random=~1|id, data=mydata))

Here is my output for ANOVA

    > summary(aov(outcome~factor(group)*factor(time)+covariate+Error(id)))

    Error: id
          Df Sum Sq Mean Sq
    factor(group)  1  4.255   4.255

    Error: Within
                       Df Sum Sq Mean Sq F value   Pr(>F)    
    factor(group)               1  94.98   94.98   57.99 1.33e-07 ***
    factor(time)                1 137.29  137.29   83.81 5.87e-09 ***
    covariate                   1  98.60   98.60   60.20 9.78e-08 ***
    factor(group):factor(time)  1  39.55   39.55   24.15 6.49e-05 ***
    Residuals                  22  36.04    1.64                     
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Here is my output for the mixed effects model

    > anova(lme(outcome~factor(group)*factor(time)+covariate, random=~1|id, data=mydata))
                       numDF denDF   F-value p-value
    (Intercept)                    1    12 1249.4127  <.0001
    factor(group)                  1    12   23.7548   4e-04
    factor(time)                   1    11  115.8036  <.0001
    covariate                      1    11   75.0274  <.0001
    factor(group):factor(time)     1    11   28.0040   3e-04

As you can see totally different. Any help would be greatly appreciated.

$\endgroup$
2
$\begingroup$

There are a few reasons as to why an ANCOVA might give you different results from an a linear mixed-effects model (LMM). ANOVAs/ANCOVAs require a very strict set of assumptions to be met for them to be appropriate for ones data, and these assumptions are rarely met, while LMMs are typically quite flexible. Due to these strict assumptions ANOVA/ANCOVAs aren't commonly used, at least not correctly.

I suspect that the main reason the two are giving very different results is that ANOVA's and ANCOVAs typically fall apart when the group sample sizes are different. Repeated measures ANOVA/ANCOVA can't cope with the fact that there are different numbers of individuals in each group within the time points measured. Mixed models can cope with this by altering the estimation for each Time × Group combinations variability.

This article goes through all the limitations of ANOVAs/ANCOVAs quite succinctly if you want to learn more about them. Let me know if you want any more clarification though!

TL;DR: ANOVAs can't cope with unbalanced groups, while LMMs can. Don't use ANOVAs/ANCOVAs.

$\endgroup$
  • 1
    $\begingroup$ Thanks for your comment, thumbs up, I figured out what it was: I had a time varying covariate. It was recorded on two separate occasions with different scores for each. When i just used the covariate from start time, it equaled. From a cursory glance of the internet, it appears that ANOVA can't handle time varying covariates at all, but if you know of anyway to get it to handle that, i would be interested. Thanks for the answer. $\endgroup$ – Huy Pham Feb 14 at 8:46
  • $\begingroup$ Glad you figured it out! What exactly is your 'outcome' measuring? Time-varying covariates are often used in survival analysis. If your outcome variable fits the requirements then perhaps this would be an avenue you could explore. $\endgroup$ – André.B 26 mins ago

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.