I'm new to both regression and multilevel modelling, and I'm having trouble with the analysis for my experiment design.

For my study, we are having subjects come in and solve 2 problems. Each problem $P$ has 2 $variants$, lets call them $A$ and $B$. Variant $A$ is the control, and variant $B$ is created by making a modification to variant $A$. Each subject receives each problem and each variant. For example, a subject could receive variant $A$ of problem 1 and variant $B$ of problem 2, or variant $B$ of problem 1 and variant $A$ of problem 2. So basically a subject won't get the same variant twice or the same problem twice.

The response variable is some performance metric $Y$. I am interested in the impact of the variant on $Y$.

The variants might have a different effect based on the problem used to create it, as some problems might inherently be more difficult than others. Also, individuals are expected to have inherent differences in their performance metric $Y$.

So far, I know that the measurement occasion will be Level 1 and participant Level 2, and that I'm going to use the treatment($variant$) as a fixed effect. But I'm not sure how to integrate the problem $P$ into the whole situation here. I'm not sure whether to use the $P$ as level 3 as follows:

$$ Y = variant + (1 | P) + (1 | participant\_id) $$

or if I should model it as an interaction

$$ Y = variant * P + (1 | participant\_id) $$

I think that using it as a level better represents the hierarchy of my data, but because I only have 2 levels for $P$, I don't think that's a large enough group size for to use it as a level. What do you think?

  • $\begingroup$ How many problems are there ? $\endgroup$ Oct 31, 2019 at 11:13
  • $\begingroup$ Only 2 problems. But I'm also planning to use this experiment design for other experiments where participants will get more problems, but the number of problems should never exceed 4 (due to time constraints). $\endgroup$ Oct 31, 2019 at 15:23

1 Answer 1


You can't model it with (1|P) if there are only 2 levels of P since you will be asking the software to estimate the variance if a variable from only 2 observations.

If you want to treat P as random then you could use (1|P:ParticipantID)

Or you could treat P as fixed, as in your 2nd example.

  • $\begingroup$ Thanks Robert! I have a similar question about incorporating time, to see if participants performing worse on the second problem due to fatigue from the first, or conversely, if they perform better on the second problem due to some kind of order effect. Because time $T$ again has 2 levels, I'm guessing incorporating it as a fixed effect would be the way to go? $\endgroup$ Oct 31, 2019 at 16:03
  • $\begingroup$ Hmm. Don't the 2 variants A and B already capture the order ? If so, a time variable could be colinear with variant. Anyway, this is a separate question so please make a new post about it. $\endgroup$ Oct 31, 2019 at 16:22
  • 1
    $\begingroup$ The variants will not always be presented in the same order, but I'll make a separate question about this. Appreciate your help! $\endgroup$ Oct 31, 2019 at 16:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.