I am trying to analyse the results from the Attrak Diff questionnaire, which is split into four 'quality' categories, and I was asking participants their opinions on three different feedback types. I got a significant p value from the one way repeated measures ANOVA so decided to run a Tukey test. The code I have for the Tukey test doesn't let me account for the second level of division though. So how can I run Tukey based on both quality and feedback, to be able to say stuff like 'participants considered the Spheres feedback to be more attractive than the Arrows feedback'? (Attractiveness is one of the categories of the Attrak Diff.) Thanks in advance!


attrakdiffdataframe <- read.csv("attrak diff.csv")

M <- lmer(value ~ quality+(1|feedback), data=attrakdiffdataframe)

M0 <- lmer(value ~ (1|feedback), data=attrakdiffdataframe)


ANOVA <- anova(M0,M)

g <- glht(M,linfct=mcp('quality'='Tukey'))
  • $\begingroup$ Just a quick comment: you have "feedback" as a random effect grouping variable, not as a fixed predictor, so you haven't modeled the fixed effect of feedback or the interaction. That's why you can't get the post-hoc tests for feedback from the model you have specified. Try M<-lmer(value ~ quality*feedback + (1|participant), data=data) and then try to do the post-hoc comparisons (emmeans package may be easier to use for that than multcomp). $\endgroup$
    – Sointu
    Dec 16, 2023 at 23:04
  • $\begingroup$ @Sointu that gives me this error: Error in list2env(data) : first argument must be a named list Does it mean I need to convert my values column to a list? $\endgroup$ Dec 17, 2023 at 1:37
  • $\begingroup$ Did you put in the name of your own dataframe into the formula, or did you literally put in data=data? If so, putting in data=attrakdiffdataframe will probably fix this. If you already did that the problem is likely in your data structure, make sure it is in long format etc. $\endgroup$
    – Sointu
    Dec 17, 2023 at 8:20
  • $\begingroup$ @Sointu I'm very embarrassed I missed that, thank you for the great spot! The sleep deprivation is starting to kick in :( I'm still not clear how to do the post-hoc comparison based on both quality and feedback. I'm happy to use whatever gets the job done package wise, but I haven't been able to find any clear examples for Tukey from emmeans. $\endgroup$ Dec 17, 2023 at 10:18
  • $\begingroup$ That's an easy mistake to make. Do you want just the main effects of quality of feedback or their interaction too? $\endgroup$
    – Sointu
    Dec 17, 2023 at 11:16

1 Answer 1


So, if you want the main effects of quality and feedback, and not the interaction, I suggest the following:

M<-lmer(value ~ quality + feedback + (1|participant), data=attrakdiffdataframe) #replace "participant" with the name of the variable that signifies your participant id


emquality<-emmeans(M, pairwise ~ quality)
emfeedback<-emmeans(M, pairwise ~ feedback) #emmeans automatically adjusts for multiple comparisons using Tukey correction when used like this.


There's a separate issue that a multilevel model may be a bit of an "overkill" for your design, you could probably do well with a single-level regression with robust standard errors, or a generalized least squares regression, but this is to get you over the current issues :)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.