I'm new to both regression and multilevel modelling, and I'm having trouble with the analysis for my experiment design.
For my study, we are having subjects come in and solve 2 problems. Each problem $P$ has 2 $variants$, lets call them $A$ and $B$. Variant $A$ is the control, and variant $B$ is created by making a modification to variant $A$. Each subject receives each problem and each variant. For example, a subject could receive variant $A$ of problem 1 and variant $B$ of problem 2, or variant $B$ of problem 1 and variant $A$ of problem 2. So basically a subject won't get the same variant twice or the same problem twice.
The response variable is some performance metric $Y$. I am interested in the impact of the variant on $Y$.
The variants might have a different effect based on the problem used to create it, as some problems might inherently be more difficult than others. Also, individuals are expected to have inherent differences in their performance metric $Y$.
So far, I know that the measurement occasion will be Level 1 and participant Level 2, and that I'm going to use the treatment($variant$) as a fixed effect. But I'm not sure how to integrate the problem $P$ into the whole situation here. I'm not sure whether to use the $P$ as level 3 as follows:
$$ Y = variant + (1 | P) + (1 | participant\_id) $$
or if I should model it as an interaction
$$ Y = variant * P + (1 | participant\_id) $$
I think that using it as a level better represents the hierarchy of my data, but because I only have 2 levels for $P$, I don't think that's a large enough group size for to use it as a level. What do you think?
