I am editing my question as it was not detailed enough. I made an (unsuccessful) shortcut. Sorry, here is the entire story.

In my experiment I test subjects' reactions to some (simulated) situations. The subject read a scenario and then an expert evaluates the subject's behavior. The evaluation ranges from 1 to 5. There are 10 different simulations, and each subject takes all of them; thus, from each subject I have 10 data points. My experiment is went on 30 days. In each day, the same 10 simulations are used. In other words, in each day, subjects and simulations are fully crossed. Each day the simulations are different.

There are 3 categories of simulations (A, B and C). The categories are, from a theoretical perspective, different one from the other. Category A is tested by 3 simulations (a1, a2, a3); B by 3 (b1, b2, b3); C by 4 (c1, c2, c3, c4). a1:c4 at day 1 are different from a1:c4 at day 2 and so on. Importantly, participants took the experiment one time only and are not allowed to participate again. That is, if participant participates on, let say day 1, he will take the 10 simulations that were on day 1. But he could never participate again.

The 3 categories are the only ones I am interested in. In that sense, I think they should be treated as fixed effects. Each category is tested/represented by some simulations. Yet, for each category there are an infinity of possible simulations and I just sampled some. In that sense simulations are random.

The only question I am interested in here, is about the effect of the subject gender on the grades. I want to control for all other parameters. My question is how to account for the simulation and category. I would like to extrapolate my results beyond participants and the simulations representing the category. Yet it is also possible that gender would interact with category or simulation.

So here is one "basic" model:

lmer(grade ~ gender + (1|subject) + (1|simulation:day), data = My_data)

Yet, this model does not account for the possibility that gender has a different effect on simulations. So here is another one trying that.

lmer(grade ~ gender + (1|subject) + (1 + gender|simulation:day), data = 

And here I get stuck. How does category play a role? Do I need to enter it as fixed effect? If yes, what about simulations? Does the following make sense?

lmer(grade ~ gender*category + (1 + category|subject) + (1 + 
gender|simulation:day), data = My_data)

Or is it better to give up the simulations, and treat category as random? But in this case, for a given day, the same category will appear several time for each subject (e.g., A will appear 3 times). Is't that a problem? As follow:

lmer(grade ~ gender + (1 + subject) + (1 + gender|category), 
data =My_data)

A final point: I have a lot of data (several thousands of participants), so convergence should not be a problem.

Thanks a lot for the help

  • 1
    $\begingroup$ What is your research question ? Are you interested in the treatment effects of stimulus ? Please edit your question to include the output of str(My_data) and the output of xtabs(~ stimulus + subject, My_data) and xtabs(~ category + subject, My_data) and xtabs(~ category + stimulus, My_data) $\endgroup$ Feb 27, 2019 at 17:35
  • $\begingroup$ You say: "for each category there are an infinity of possible simulations and I just sampled some." Were any of the individual simulations used more than once? Or did you have 3N (where N is the number of participants) separate simulations that were in Category A, for example, and similarly for Categories B and C? How many simulations were there overall for each of the Categories? $\endgroup$
    – EdM
    Feb 27, 2019 at 21:04
  • $\begingroup$ Per your update, I believe we are thinking about different uses for the terms random and fixed effects - see stats.stackexchange.com/questions/4700/… for a discussion and on that page: statmodeling.stat.columbia.edu/2005/01/25/why_i_dont_use. I am not entirely sure how to do what you want within LME4. But in general, if you are asking about a linear mixed effect model, like with lmer( ), people will most likely think you are trying to find effects like I specified below. $\endgroup$ Feb 27, 2019 at 22:33

5 Answers 5


A couple of points:

  • If I understood correctly, your response variable is a grade, ranging from 1 to 5. This is an ordinal variable with relatively few levels. Hence, assuming a normal distribution for the residuals may not be appropriate. You could consider a mixed model for ordinal data instead, such as the continuation ratio model.
  • You are right that subject and simulation seem to be fully crossed factors. Hence, a possible model to consider is:

    grade ~ category * gender + (1 | subject) + (1 | simulation) 
  • $\begingroup$ I see that including category as fixed and simulation as random is ok for you. Some comments: First, (time [ subject) may not be appropriate here as each participant took the experiment only one time. If you were on day 1, you took all 10 simulations, but you could not take the experiment another time. Second, as my only interest here is in the effect of gender, I think I should account for the fact that in some simulations/category, gender might have an effect and in other not, so I may include (1 + gender | simulation). Would you like to edit your answer? $\endgroup$
    – Rtist
    Feb 28, 2019 at 8:38
  • $\begingroup$ @Rtist so different subjects are used each day? $\endgroup$ Feb 28, 2019 at 8:44
  • $\begingroup$ exactly. I made it clear now in the question. $\endgroup$
    – Rtist
    Feb 28, 2019 at 9:35
  • $\begingroup$ @Rtist I made an update. $\endgroup$ Feb 28, 2019 at 19:01

If you believe the different categories vary in terms of their measurement and you want to test that, you need to model the category as a random intercept with (1|category).

However, I think we need more information as to what you are actually looking to decide. For example, are you wondering if each person measures them different? Or if each stimulus is measured differently?

  • $\begingroup$ The category could vary. Yet, there are only 3 of them (each one being measure with different stimuli) and theoretically there are no other categories. So it would be difficult to model them as random. $\endgroup$
    – Rtist
    Feb 27, 2019 at 22:40

It seems that in the model:

lmer(measure ~ category + (1|subject) + (1|stimulus), data = My_data)

category is being used to denote the levels of stimulus. As such, this does not make sense, since category is not an actual variable.

Even though stimulus is random in the sense that it has (presumably) been randomly assigned to each subject, this does not mean that is should be included as a random effect - unless there is no interest in the treatment effect of stimulus. In that case, you would simply be partitioning variance into the subject level and the stimulus level.

It seems more likely that you are in fact interested in the associations between each level of the stimulus and the outcome - that is, you are interested in the treatment effect and therefore the model should be of the form:

measure ~ stimulus + (1|subject)
  • 2
    $\begingroup$ Interesting perspective, Robert! If the 10 stimuli are the only ones @Rtist is interested in, then your suggested approach makes sense. But if he only included the 10 stimuli in the study because he is interested in generalizing the findings of the study to all stimuli represented by these 10 items, then stimulus should be treated as a random grouping factor (assuming there are multiple observations for at least some of all the subject by stimulus combinations). It really all depends on the purpose of the study - if we don't know the purpose, we can only speculate about the model set up. $\endgroup$ Feb 27, 2019 at 17:01
  • $\begingroup$ @IsabellaGhement there are only 3 levels of stimulus, so fitting random effects for it and estimating a variance would not be advisable in my opinion. $\endgroup$ Feb 27, 2019 at 17:09
  • 1
    $\begingroup$ But the question says each subject sees 10 stimuli and that these 10 stimuli can be categorized into 3 different categories? I am not suggesting stimulus category (3 levels) could be treated as a random grouping factor, but rather stimulus (10 levels). However, if there aren't repeated values of measure for each subject by stimulus combination OR if the research question calls for it (i.e., the 10 stimuli are the only ones @Rtist wants to learn something about), then the option you suggested would work. $\endgroup$ Feb 27, 2019 at 17:24
  • 1
    $\begingroup$ @IsabellaGhement perhaps I misunderstood the question. I read it that stimulus has 3 levels, not 10. Anyway, we can wait for clarification from @Rtist $\endgroup$ Feb 27, 2019 at 17:31

In your setting, subject and stimulus seem to be fully crossed random grouping factors - since each subject sees each stimulus and (I am assuming) you are using the subjects and stimuli included in your studies to represent all the subjects and all the stimuli you wish to generalize your study findings to.

The key word here is grouping - for your model to be a linear mixed effects model (lmer), each subject by stimulus combination should act like a container which holds together a group of values for your measure outcome. All the values of measure that belong to the same container are more similar to each other than values that belong to different containers, as they are subjected to the same subject-level and stimulus-level influences (presuming these influences are constant over time).

The group of values in a specific container could arise, for instance, if you record the value of measure at several time points for each subject by stimulus combination, or under two or more different conditions, etc.

If you only have one value of measure per subject by stimulus combination, then you're dealing with a linear model (lm). There is no grouping of observations according to each subject per stimulus combination, so there are no random grouping factors which means there aren't any effects that can vary randomly across combinations of levels of the grouping factors (i.e., random effects). If there aren't any random effects, there can't be a mixed effects model, as such a model would require both fixed and random effects to be part of it!

If you do have multiple values of measure per container (i.e., subject by stimulus combination), then your model can include subject-level predictors (e.g., subject gender, subject age) and/or stimulus-level predictors (e.g., stimulus category).


One important point is what you mean by "a1:c4 at day 1 are different from a1:c4 at day 2 and so on". If this means that the a1 from day 1 is not more linked to a1 from day 2 that to a2 from day 2, then you don't have 10 simulations, but 10x30= 300 simulations (90 from category A, 90 from category B and 120 from category C). You should name them differently (e.g a1day1, a2day1, ..., a3day30, b1day1, ...) and call this column e.g. AllSim. Then AllSim is you random effect column and you can construct models from it. The fact that subjects see only 10 of these 300 simulations is not a problem and is handled automatically by lme4. My guess is that the most interesting models will be

grade ~ category * gender + (1 | subject) + (1 | AllSim) 

or if you put the maximal random slopes

grade ~ category * gender + (category | subject) + (gender | AllSim) 

or even more complex models if you beleive e.g. that subjects get better and better with practice over they 10 simulations.

Edit: I read it too quickly, and actually your simulation:day, when put on the right of the "|" is exactly the same as my AllSim, so for the basic random effects, your 3 first models are totally right (and the two first ones correspond to my two models). The choice between these models should be either data-based or theory-driven. And as Box said, "all models are wrong, but some are useful."


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.