I am running a post-hoc analysis on the data collected during an experiment in which 15 unique stimuli were presented to participants. Having run a least squares regression using the lm() function in R I have found significant results for a subset of the data including 90 observations from 6 participants with two continuous variables and their interaction.
Taking advice from an article by Judd, Westfall & Kenny (2012) I attempted to use a combination of the lmer() function found in the lme4 package in combination with a Kenward-Roger approximation through the KRmodcomp() function in the pbkrtest package (see the appendix in the article) in order to control for random effects:
lmerlmer(Prediction_Difference_Scale~Diff_AWD_LRTI_End_Scale*Diff_AWD_BD_End_Scale + (1|Unique_ID) + (Diff_AWD_LRTI_End_Scale*Diff_AWD_BD_End_Scale|Block),data=Data)
The first variable after the DV is the fixed effect, the second variable in parentheses indicates that the intercept is random with respect the unique stimuli (Unique_ID) and the third variable in parentheses indicates that both the intercept and the Condition slopes are random with respect to participant (Block) and that a covariance between the effects should be estimated.
When running the lmer() function I get the following error message:
Error in checkNlevels(reTrms$flist, n = n, control) :
number of levels of each grouping factor must be < number of observations
This is obviously because the number of observations equal the number of unique stimuli.
The function works when excluding the (1|Unique_ID) random effect, which if I understand correctly is the same as carrying out a 'by stimulus' analysis. However, the authors warn against this by stating: "Conceptually, a significant by-participant result suggests that experimental results would be likely to replicate for a new set of participants, but only using the same sample of stimuli. A significant by-stimulus result, on the other hand, suggests that experimental results would be likely to replicate for a new set of stimuli, but only using the same sample of participants. However, it is a fallacy to assume that the conjunction of these two results implies that a result would be likely to replicate with simultaneously new samples of both participants and stimuli."
I would like to control for the random effects of both stimuli and participants, but I am unsure how to proceed?
The article can be accessed here: http://jakewestfall.org/publications/JWK.pdf
To clarify the question regarding the 15 unique stimuli, this is 15 unique stimuli per participant, meaning the sample of 90 observations consists of 6 participants. The stimuli for all of the 90 observations are unique however.
I suppose what my question boils down to is whether there is even a need to include the (1|Unique_ID) 'variable' in the function formula as there is no error dependence between any of the stimuli?
