I've computed an ANOVA with one between-subjects factor (2 groups each including 26 participants) and 2 within-subjects factors (item type with 3 levels and emotion with 2 levels) in ezANOVA. Therefore, my data is ordered in a long-format in which all conditions for each participant are listed underneath each other:

ID     group    itemType    emotion     dependentVariable
p001      A        type1    negative                 3.88
p001      A        type1    neutral                  2.34
p001      A        type2    negative                 5.21
p001      A        type2    neutral                 10.00

When I am now computing post-hoc t-tests, e.g. the difference between two item types in a specific group regardless of the emotionality of the items, should I then keep working in this long-format? Or should I change into a wide-format which includes the average over the non-interesting factor, in this case emotionality? I am wondering if in case of the long-format, R might treat the levels of the non-interesting factor as two independent observations and that this computation might then be statistically incorrect?

In case of the long-file, I used this code:

t.test(x = long_file$dependentVariable[long_file$delay=="A" & long_file$itemType=="type1"], 
       y = long_file$dependentVariable[long_file$delay=="A" & long_file$itemType=="type2"], 
       paired = TRUE)

resulting in t=2.43, df=51, p=0.02

I used this code to analyze the dependent variable as average of the non-interesting factor in a wide-format:

t.test(x = wide_file$dependentVariable_type1_averageOverEmotions[wide_file$group=="A"], 
       y = wide_file$dependentVariable_type2_averageOverEmotions[wide_file$group=="A"], 
       paired = TRUE)

resulting in t=2.38, df=25, p=0.03

So, in this case, results don't change a lot regarding the statistical significance, but using the long-format doubles the degrees of freedom.


2 Answers 2


Since you have a rather complex model with both between factors and within factors, you are better off using a post-hoc analysis that takes the model into account, rather than using a series of t-tests.

I assume you are using R. One approach would be to use the emmeans package, but I don't think ezAnova objects are supported cran.r-project.org/web/packages/emmeans/vignettes/models.html.

You may want to refit your model with a more-supported package, like lme4 or nlme.

  • $\begingroup$ Thanks for the response. I ran other post-hoc analyses and those t-tests are just one part of it. I did try linear mixed models, but I decided to use ANOVAs as I only have categorical factors, one with 3(!) levels, and the ANOVA-results are therefore much easier to interpret. I already found my answer and will always use the mean of an uninteresting factor in case of t-tests. Thanks anyways! $\endgroup$
    – valid
    Feb 17, 2022 at 16:26
  • $\begingroup$ Anova is just a form of general linear model. You should be able to get an anova-like output for models fit with common packages. ... In any case, it's worth your while to learn to use packages like emmeans or multcomp for post-hoc comparisons. They represent flexible and modern approaches, and reflect the fitted model under consideration. $\endgroup$ Feb 17, 2022 at 16:58
  • $\begingroup$ I will have a look into emmeans and multcomp, thanks for the suggestion! $\endgroup$
    – valid
    Feb 18, 2022 at 12:12

My solution for post-hoc tests for ANOVAs computed with ezANOVA: If a variable that contains factor levels in the long format that are not compared in the t-test using t.test in R, then the degrees of freedom are overestimated (doubled in this case). Therefore, in this case, the variable should first be averaged over the non-interesting factor before computing the post-hoc t-test.

For future ANOVAs, I would rather use other functions such as aov which enable post-hoc tests with functions that take the model into account and also correct for multiple comparisons such as emmeans.

  • 1
    $\begingroup$ What is seriously missing here is any conception of what is the underlying statistical model. It is really important to understand that, otherwise one is following a blind path. $\endgroup$
    – Russ Lenth
    Feb 19, 2022 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.