1
$\begingroup$

I am running a post-hoc analysis on the data collected during an experiment in which 15 unique stimuli were presented to participants. Having run a least squares regression using the lm() function in R I have found significant results for a subset of the data including 90 observations from 6 participants with two continuous variables and their interaction.

Taking advice from an article by Judd, Westfall & Kenny (2012) I attempted to use a combination of the lmer() function found in the lme4 package in combination with a Kenward-Roger approximation through the KRmodcomp() function in the pbkrtest package (see the appendix in the article) in order to control for random effects:

lmer(Prediction_Difference_Scale~Diff_AWD_LRTI_End_Scale*Diff_AWD_BD_End_Scale + (1|Unique_ID) + (Diff_AWD_LRTI_End_Scale*Diff_AWD_BD_End_Scale|Block),data=Data)

The first variable after the DV is the fixed effect, the second variable in parentheses indicates that the intercept is random with respect the unique stimuli (Unique_ID) and the third variable in parentheses indicates that both the intercept and the Condition slopes are random with respect to participant (Block) and that a covariance between the effects should be estimated.

When running the lmer() function I get the following error message:

Error in checkNlevels(reTrms$flist, n = n, control) : 
  number of levels of each grouping factor must be < number of observations

This is obviously because the number of observations equal the number of unique stimuli.

The function works when excluding the (1|Unique_ID) random effect, which if I understand correctly is the same as carrying out a 'by stimulus' analysis. However, the authors warn against this by stating: "Conceptually, a significant by-participant result suggests that experimental results would be likely to replicate for a new set of participants, but only using the same sample of stimuli. A significant by-stimulus result, on the other hand, suggests that experimental results would be likely to replicate for a new set of stimuli, but only using the same sample of participants. However, it is a fallacy to assume that the conjunction of these two results implies that a result would be likely to replicate with simultaneously new samples of both participants and stimuli."

I would like to control for the random effects of both stimuli and participants, but I am unsure how to proceed?

The article can be accessed here: http://jakewestfall.org/publications/JWK.pdf

To clarify the question regarding the 15 unique stimuli, this is 15 unique stimuli per participant, meaning the sample of 90 observations consists of 6 participants. The stimuli for all of the 90 observations are unique however.

I suppose what my question boils down to is whether there is even a need to include the (1|Unique_ID) 'variable' in the function formula as there is no error dependence between any of the stimuli?

Improve this question
$\endgroup$
3
  • $\begingroup$ But you say you have 90 observations and 15 unique stimuli; would you clarify? $\endgroup$
    – Aaron - mostly inactive
    Commented Jul 30, 2014 at 17:35
  • $\begingroup$ in addition to what Aaron says, when I read the first sentence, I thought stimuli are the treatments which you might want to treat as fixed. But you obviously want something different. Perhaps some context will help. $\endgroup$
    – qoheleth
    Commented Jul 31, 2014 at 1:53
  • $\begingroup$ to follow up on Aaron's question; are there really 90 distinct Unique_ID values in your sample of 90 observations? (The results of summary(Data) would be helpful ...) You can use lmerControl to override the test and fit the model anyway -- your among-Unique_ID and residual variances will be jointly unidentifiable, but that likely (??) won't affect the estimation of the other terms ... (papering over cracks in this way rather than understanding the analysis more thoroughly might be a bad approach, though) $\endgroup$
    – Ben Bolker
    Commented Aug 2, 2014 at 16:00

1 Answer 1

Reset to default
2
$\begingroup$

This design does not involve crossed random factors; rather, stimuli here are nested in participants. So the advice of our paper, which is concerned with crossed random factors, does not apply to this kind of design.

Furthermore, because each stimulus is observed only once, stimulus effects cannot be estimated at all (as hinted already by @BenBolker) because the stimulus effects cannot be distinguished from the residual terms.

The upshot of all this is that the correct model to estimate for this (nested) design is one that only includes random effects for participants. Hope this helps.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.