I conducted an experiment in which I timed participants completing various tasks, and I'm trying to figure out what factors influenced completion time and if that influence was significant (using R).

I have followed quite a few R tutorials on ANOVA testing, but my data has a few nuances: it's a repeated measures / within subjects design (9 participants, 74 observations total); each participant did multiple tasks with and without training (my main factor of interest); and I have incomplete block design because not all participants completed all tasks. However, every task that was completed was done with & without training.

The goal is to assess whether or not training helps save time for task completion, while accounting for the different tasks and subjects as sources of variance. The data looks like this:

> df[31:40,]
# A tibble: 10 × 5
   TaskPerformer YearsOfExperience TaskID TrainedForTask TotalSeconds
   <fct>                     <dbl> <fct>  <fct>                 <dbl>
 1 Tester5                      15 4      TRUE                   230.
 2 Tester9                      16 4      TRUE                   116.
 3 Tester3                      20 5      TRUE                   305.
 4 Tester3                      20 1      TRUE                   302.
 5 Tester2                      12 1      TRUE                   293.
 6 Tester9                      16 1      TRUE                   272.
 7 Tester3                      20 3      TRUE                   251.
 8 Tester7                      15 2      FALSE                  190.
 9 Tester7                      15 4      FALSE                  159.
10 Tester3                      20 1      FALSE                  149.

The block design:

> xtabs(~ TaskPerformer + TaskID + PreppedForTask, data = df)
, , PreppedForTask = FALSE

TaskPerformer 1 2 3 4 5
      Tester1 0 1 1 1 1
      Tester2 1 1 1 1 1
      Tester3 1 1 1 1 1
      Tester4 0 1 1 1 0
      Tester5 0 1 1 1 1
      Tester6 0 1 1 1 1
      Tester7 0 1 1 1 0
      Tester8 1 0 1 1 1
      Tester9 1 1 1 1 1

"PreppedForTask = TRUE" is similar.

An incomplete list of what I've tried:

  • afex::aov_car(TotalSeconds ~ TrainedForTask*TaskPerformer + Error(TaskPerformer), data=df). This doesn't work because the testers do multiple tasks. Error msg: "Following ids are in more than one between subjects condition: Tester1, ... Tester9". Also, it doesn't take into account the TaskID. Some tasks are more difficult than others and could impact time taken.
  • 3-way ANOVA with the TaskID and TrainedForTask as 2 'within-subjects' factors + TaskPerformer as the 1 'between-subjects' factor. First of all, I'm not even sure that's the right set up, but it doesn't work because of the incomplete block design. I could do a 1- or 2-way ANOVA, but if I just look at TrainedForTask and TotalSeconds, the training doesn't have a significant impact - I need to know how TaskPerformer and TaskID affect the results.
  • Based on this question, I tried: anova(lmer(TotalSeconds ~ TrainedForTask + TaskID + (1|TaskPerformer), data=df)) which ran without errors but shows no statistical significance, which is surprising because a paired, one-sided t-test (assuming training improves task time) does have p-val < 0.05. And similar to the person who asked that question, I am not sure I should be treating TaskPerformer like this or if it should be included as a main effect.

I'm happy to share more detail if needed. I have seen lots of similar questions but nothing that has this combo of multi-factor, incomplete block design, and repeated measures with multiple tasks per subject. Thanks in advance for your thoughts.


1 Answer 1


I haven't tried using lmer inside the anova command, so not sure about that, but it seems to me you could just use the regular lmer function. I wouldn't put the participant (TaskPerformer refers to the person conducting the tasks, right?) in as fixed effect (which you probably mean by 'main effect') because you are using multilevel approach to control for possible within-participant clustering (right?). Using participant ID as random effect does this nicely, and I don't see any advantage in using it as a fixed effect. I'd consider also changing TaskID into a random effect, because you say "... while accounting for the different tasks and subjects as sources of variance. "

So, you could try

model<-lmer(Totalseconds ~ (1|TaskPerformer)+(1|TaskID)+ TrainedForTask, data=df)

This gives you the effect of TrainedForTask on TotalSeconds while dealing with the Performer and Task-specific variance in Totalseconds.

  • $\begingroup$ Hi @Sointu, thanks for the thoughts. Yes, the TaskPerformer is the participant completing the tasks. Since the different participants have varying levels of experience / expertise, I want to account for that (which I think I am doing with (1|TaskPerformer) in my lmer example as well as yours). I think you're right about the TaskID too - that's helpful. I'm still curious about whether or not an ANOVA is appropriate here, and if not, why not. $\endgroup$
    – mercer721
    Dec 28, 2022 at 19:01
  • $\begingroup$ I think repeated measures ANOVA is appropriate in principle in your design, but it's not recommended if you have different number of observations at different levels of your factor, as you do. I'd personally use lmer in your case. But as I mentioned, I'm not sure what anova(lmer(... does. This reply may be useful: stats.stackexchange.com/questions/71914/… $\endgroup$
    – Sointu
    Dec 30, 2022 at 9:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.