I'm new to both regression and multilevel modelling, and I'm having trouble with the analysis for my experiment design.

For my study, we are having subjects come in and solve 2 problems. Each problem $P$ has 2 $variants$, lets call them $A$ and $B$. Variant $A$ is the control, and variant $B$ is created by making a modification to variant $A$. Each subject receives each problem and each variant. For example, a subject could receive variant $A$ of problem 1 and variant $B$ of problem 2, or variant $B$ of problem 1 and variant $A$ of problem 2. So basically a subject won't get the same variant twice or the same problem twice.

The response variable is some performance metric $Y$. I am interested in the impact of the variant on $Y$.

The variants might have a different effect based on the problem used to create it, as some problems might inherently be more difficult than others. Also, individuals are expected to have inherent differences in their performance metric $Y$.

So far, I know that the measurement occasion will be Level 1 and participant Level 2, and that I'm going to use the treatment($variant$) as a fixed effect. But I'm not sure how to integrate the problem $P$ into the whole situation here. I'm not sure whether to use the $P$ as level 3 as follows:

$$ Y = variant + (1 | P) + (1 | participant\_id) $$

or if I should model it as an interaction

$$ Y = variant * P + (1 | participant\_id) $$

I think that using it as a level better represents the hierarchy of my data, but because I only have 2 levels for $P$, I don't think that's a large enough group size for to use it as a level. What do you think?

  • $\begingroup$ How many problems are there ? $\endgroup$ – Robert Long Oct 31 '19 at 11:13
  • $\begingroup$ Only 2 problems. But I'm also planning to use this experiment design for other experiments where participants will get more problems, but the number of problems should never exceed 4 (due to time constraints). $\endgroup$ – Devjeet Roy Oct 31 '19 at 15:23

You can't model it with (1|P) if there are only 2 levels of P since you will be asking the software to estimate the variance if a variable from only 2 observations.

If you want to treat P as random then you could use (1|P:ParticipantID)

Or you could treat P as fixed, as in your 2nd example.

  • $\begingroup$ Thanks Robert! I have a similar question about incorporating time, to see if participants performing worse on the second problem due to fatigue from the first, or conversely, if they perform better on the second problem due to some kind of order effect. Because time $T$ again has 2 levels, I'm guessing incorporating it as a fixed effect would be the way to go? $\endgroup$ – Devjeet Roy Oct 31 '19 at 16:03
  • $\begingroup$ Hmm. Don't the 2 variants A and B already capture the order ? If so, a time variable could be colinear with variant. Anyway, this is a separate question so please make a new post about it. $\endgroup$ – Robert Long Oct 31 '19 at 16:22
  • 1
    $\begingroup$ The variants will not always be presented in the same order, but I'll make a separate question about this. Appreciate your help! $\endgroup$ – Devjeet Roy Oct 31 '19 at 16:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.