I am evaluating an analysis of an experiment in which each participant was shown 5 pairs of stimuli which represented options that participants could choose between - call the two options in each pair Choice 0 and Choice 1. Each participant chose either Choice 0 or Choice 1 (which were presented in random order) for each stimulus. In addition, there were also two between-subjects conditions:
- For a given participant, either Choice 0 always had Property A (and Choice 1 did not), or Choice 1 always had Property A (and Choice 0 did not).
- For a given participant, either Choice 0 always had Property B (and Choice 1 did not), or Choice 1 always had Property B (and Choice 0 did not).
The researchers are trying to predict participants' five choices (0 or 1 in each case) using the following logistic regression model:
Choice ~ Intercept + Stimulus + PropertyA + PropertyB + PropertyA*PropertyB
- 'Stimulus' has 5 levels corresponding to the 5 pairs of stimuli and is dummy/treatment-coded, they have arbitrarily chosen the first level as the reference level.
- PropertyA is coded 1 if Choice 1 had Property A, 0 otherwise.
- PropertyB is coded 1 if Choice 1 had Property B, 0 otherwise.
My initial impulse is to suggest that they should re-run the analysis using effect coding for Stimulus since they have no reason for any particular value of this factor to serve as the reference level, and that furthermore, because each participant is making 5 choices which are likely correlated, it would be more appropriate to use a mixed effects model with a random intercept and random slope for Stimulus, e.g.
Choice ~ Intercept + (1 + Stimulus | ID) + PropertyA + PropertyB + PropertyA*PropertyB
However, given the particular details of this experiment it is possible that any given participant's responses on the five stimuli that they are presented with were only weakly correlated. If so, then is it still important for them to run this as a mixed effects model? For example, if Stimulus has a VIF of, say, less than 3 in their current model, should I still insist that they re-run it with a random intercept and random slope for Stimulus?
Any other important critiques (of either the researcher's analysis or of my planned response to it) would be welcome.