5
$\begingroup$

A set of subjects from two groups are requested to perform a test after a sleep session of type A and then after a sleep session of type B.

My data frame pdf has then an independent variable Acc (accuracy) which contains the accuracy score (percentage of correct responses) of each subject after each session type. I am interested in the effect of Group and SessionType in the accuracy score. Since the same subjects are tested on each of the session types, a repeated measures two- way ANOVA is needed, or alternatively a linear mixed-effects model. For flexibility, I chose the latter.

I have fitted a linear mixed-effects model using the lmer package:

model <- lmer(Acc ~ Group * SessionType + (1 | Subject), pdf, REML = FALSE)

After fitting the model, I was curious to see what the ANOVA test would tell me. So I used the anova function from the stats package over the linear model:

anov <- anova(model)

I expected the linear mixed-effects and the ANOVA to tell me more or less the same thing. Although one speaks of differences in mean and the other of linear coefficients, they should be (to my understanding) essentially the same. However, as you can see in the image below, they show quite different results.

tab_model(anov, model, dv.labels=c("Accuracy ANOVA", "Accuracy LME"))

enter image description here

In particular, the linear model concludes that there's no reason to believe the coefficient associated to the Group variable is different from zero, while the ANOVA model finds a significant difference between the means of both groups.

Am I misunderstanding something? How should these results be interpreted?

Edit: I include a summary of the model and of the data frame.

> str(pdf)

tibble [88 × 8] (S3: tbl_df/tbl/data.frame)
 $ Subject    : Factor w/ 48 levels "17","18","19",..: 1 1 2 2 3 3 4 4 5 5 ...
 $ SessionType: Factor w/ 2 levels "BL","SWD": 1 2 1 2 1 2 1 2 1 2 ...
 $ Group      : Factor w/ 2 levels "HC","MDD": 1 1 2 2 2 2 2 2 2 2 ...
 $ ErrsOfCom  : int [1:88] 0 0 0 0 2 4 0 0 3 0 ...
 $ ErrsOfOm   : int [1:88] 7 12 13 12 9 7 13 8 12 10 ...
 $ TotalErrors: int [1:88] 7 12 13 12 11 11 13 8 15 10 ...
 $ Acc        : num [1:88] 0.912 0.85 0.838 0.85 0.863 ...
 $ AvgRT      : num [1:88] 70.7 49.5 45.6 49.1 60.7 ...

> summary(model)
Linear mixed model fit by REML. t-tests use Satterthwaite's method ['lmerModLmerTest']
Formula: Acc ~ Group * SessionType + (1 | Subject)
   Data: pdf

REML criterion at convergence: -291.1

Scaled residuals: 
     Min       1Q   Median       3Q      Max 
-2.09564 -0.50489 -0.05255  0.48803  1.77343 

Random effects:
 Groups   Name        Variance  Std.Dev.
 Subject  (Intercept) 0.0006677 0.02584 
 Residual             0.0008636 0.02939 
Number of obs: 85, groups:  Subject, 45

Fixed effects:
                         Estimate Std. Error        df t value Pr(>|t|)    
(Intercept)              0.865222   0.010335 73.483602  83.722   <2e-16 ***
GroupMDD                -0.007741   0.012758 72.880478  -0.607   0.5459    
SessionTypeSWD           0.026002   0.011174 42.944520   2.327   0.0247 *  
GroupMDD:SessionTypeSWD -0.021414   0.013696 42.113553  -1.563   0.1254    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
            (Intr) GrpMDD SsTSWD
GroupMDD    -0.810              
SessnTypSWD -0.566  0.459       
GrMDD:STSWD  0.462 -0.563 -0.816
Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Something seems off here. This isn't the reason for the ANOVA differences - Greg Snow has explained that in his answer, but according to the table in the question, the random intercept and residual error are both estimated to have zero variance, which can't be right. Did the model converge properly ? $\endgroup$
    – Robert Long
    Commented Nov 25, 2023 at 19:26
  • $\begingroup$ Re-running the model with REML = TRUE gives a REML criterion at convergence of -289.5, with only negligible differences in the estimated coefficients and respective p-values. $\endgroup$
    – lafinur
    Commented Nov 25, 2023 at 20:10
  • $\begingroup$ Is it plausible that, because the independent variable is a percentage (ranges from 0 to 1), the variance induced by the random factors is so small that it rounds to zero? The summary of the model gives random effects standard deviation of 0.03 and 0.02 for Subject and residuals, respectively, so the variance of the random factors is indeed very negligible. $\endgroup$
    – lafinur
    Commented Nov 25, 2023 at 20:27
  • $\begingroup$ Yes, that’s plausible. Please edit the question and add the output of ‘sumnary(model)’, ‘str(pdf’) and ‘summary(pdf)’ $\endgroup$
    – Robert Long
    Commented Nov 25, 2023 at 21:44
  • $\begingroup$ I have included what you suggested. Thank you for taking the time to contribute, as you may have inferred I am rather new to statistics. $\endgroup$
    – lafinur
    Commented Nov 25, 2023 at 23:33

2 Answers 2

Reset to default
11
$\begingroup$

The anova function returns the sequential tests on the model (the aov function would be used if you wanted to compare to traditional analysis of variance).

In other words, the ANOVA $p$ value for group is testing if group has any effect on accuracy, ignoring session type and the interaction, while the group $p$ value from the lme summary is testing if group has an effect after adjusting for the session type and interaction. Very different questions being asked. The $p$ values for the interaction are very similar, because they are asking the same question about whether the interaction is significant given everything else (the ANOVA test for session type adjusts for group since it is above it in the list, but not for the interaction which is below it).

These sequential tests were used much more often in the days when analyses took a long time and things could be ordered to answer a series of pre-planned potentially interesting questions. But with modern computers we don't use the sequential tests as much because we can run multiple different models and compare them directly, often in just a few seconds (the other use of the anova function is to compare $2$ (or sometimes more) models in a full-reduced model test).

Improve this answer
$\endgroup$
2
  • $\begingroup$ I see. So while the ANOVA is indeed taking into account the fact that there are repeated measurements, it doesn't adjust for Group and SessionType (except when evaluating this specific interaction). Could you please clarify in what sense using the aov function would be different? Say, if I called anov <- aov(Acc ~ Group * SessionType + Error(Subject), pdf), should I expect to see something different from the ANOVA in my question? $\endgroup$
    – lafinur
    Commented Nov 25, 2023 at 19:29
  • 1
    $\begingroup$ @lafinur, If your residuals are normally distributed and everything is balanced, then the results of aov and lmer should be pretty similar. But aov uses sums of squares and lmer uses maximum likelihood, so they can differ when data is non-normal and/or unbalanced. $\endgroup$
    – Greg Snow
    Commented Nov 25, 2023 at 19:39
5
$\begingroup$

First, apart from Greg's excellent note about sequential vs. simultaneous tests, I wonder why you think the results should be more or less the same.

And for your final question:

How should these results be interpreted?

I would say that the ANOVA result should not be interpreted, no matter which function for them you use. As you noted, you violated the assumptions of the linear model because the errors are not independent. So, don't do the wrong test.

Finally, it would be good to show the coefficients from both models, not just the p values. Comparing p values isn't very useful.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for your answer. However, I don't understand why repeated measures ANOVA should not be used. I thought it was a rather standard test in this kind of situations (comparing before-after treatment, for example). $\endgroup$
    – lafinur
    Commented Nov 25, 2023 at 19:40
  • $\begingroup$ Incidentally, when did I say the errors were not independent? $\endgroup$
    – lafinur
    Commented Nov 25, 2023 at 19:55
  • $\begingroup$ Your code runs a regular ANOVA not an RM ANOVA. But MLMs are preferred, nowadays, because they make fewer assumptions. And, regardless of what you said, the errors are not independent when you have repeated measures. $\endgroup$
    – Peter Flom
    Commented Nov 25, 2023 at 20:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.