Appropriate ANOVA test for mixed factors, incomplete block design?

Question

I conducted an experiment in which I timed participants completing various tasks, and I'm trying to figure out what factors influenced completion time and if that influence was significant (using R).

I have followed quite a few R tutorials on ANOVA testing, but my data has a few nuances: it's a repeated measures / within subjects design (9 participants, 74 observations total); each participant did multiple tasks with and without training (my main factor of interest); and I have incomplete block design because not all participants completed all tasks. However, every task that was completed was done with & without training.

The goal is to assess whether or not training helps save time for task completion, while accounting for the different tasks and subjects as sources of variance. The data looks like this:

> df[31:40,]
# A tibble: 10 × 5
   TaskPerformer YearsOfExperience TaskID TrainedForTask TotalSeconds
   <fct>                     <dbl> <fct>  <fct>                 <dbl>
 1 Tester5                      15 4      TRUE                   230.
 2 Tester9                      16 4      TRUE                   116.
 3 Tester3                      20 5      TRUE                   305.
 4 Tester3                      20 1      TRUE                   302.
 5 Tester2                      12 1      TRUE                   293.
 6 Tester9                      16 1      TRUE                   272.
 7 Tester3                      20 3      TRUE                   251.
 8 Tester7                      15 2      FALSE                  190.
 9 Tester7                      15 4      FALSE                  159.
10 Tester3                      20 1      FALSE                  149.

The block design:

> xtabs(~ TaskPerformer + TaskID + PreppedForTask, data = df)
, , PreppedForTask = FALSE

             TaskID
TaskPerformer 1 2 3 4 5
      Tester1 0 1 1 1 1
      Tester2 1 1 1 1 1
      Tester3 1 1 1 1 1
      Tester4 0 1 1 1 0
      Tester5 0 1 1 1 1
      Tester6 0 1 1 1 1
      Tester7 0 1 1 1 0
      Tester8 1 0 1 1 1
      Tester9 1 1 1 1 1

"PreppedForTask = TRUE" is similar.

An incomplete list of what I've tried:

• afex::aov_car(TotalSeconds ~ TrainedForTask*TaskPerformer + Error(TaskPerformer), data=df). This doesn't work because the testers do multiple tasks. Error msg: "Following ids are in more than one between subjects condition: Tester1, ... Tester9". Also, it doesn't take into account the TaskID. Some tasks are more difficult than others and could impact time taken.
• 3-way ANOVA with the TaskID and TrainedForTask as 2 'within-subjects' factors + TaskPerformer as the 1 'between-subjects' factor. First of all, I'm not even sure that's the right set up, but it doesn't work because of the incomplete block design. I could do a 1- or 2-way ANOVA, but if I just look at TrainedForTask and TotalSeconds, the training doesn't have a significant impact - I need to know how TaskPerformer and TaskID affect the results.
• Based on this question, I tried: anova(lmer(TotalSeconds ~ TrainedForTask + TaskID + (1|TaskPerformer), data=df)) which ran without errors but shows no statistical significance, which is surprising because a paired, one-sided t-test (assuming training improves task time) does have p-val < 0.05. And similar to the person who asked that question, I am not sure I should be treating TaskPerformer like this or if it should be included as a main effect.

I'm happy to share more detail if needed. I have seen lots of similar questions but nothing that has this combo of multi-factor, incomplete block design, and repeated measures with multiple tasks per subject. Thanks in advance for your thoughts.

Answer 1

I haven't tried using lmer inside the anova command, so not sure about that, but it seems to me you could just use the regular lmer function. I wouldn't put the participant (TaskPerformer refers to the person conducting the tasks, right?) in as fixed effect (which you probably mean by 'main effect') because you are using multilevel approach to control for possible within-participant clustering (right?). Using participant ID as random effect does this nicely, and I don't see any advantage in using it as a fixed effect. I'd consider also changing TaskID into a random effect, because you say "... while accounting for the different tasks and subjects as sources of variance. "

So, you could try

model<-lmer(Totalseconds ~ (1|TaskPerformer)+(1|TaskID)+ TrainedForTask, data=df)

This gives you the effect of TrainedForTask on TotalSeconds while dealing with the Performer and Task-specific variance in Totalseconds.