Is there a way to ask for "what changed between experiments"?

Question

I have performed an experiment in which participants perform a task multiple times, the outcome being a binary "has succeeded or not".

Two cohorts have performed the same experiment in two conditions ($C_A$ and $C_B$) where the only difference between cohorts was in which condition they started.

For every sample the success and a range of parameters were recorded, some of them fixed (like the age or arm length of the participants) some of them a result of the experiment (like a reaction time for that sample).

Up to this point I have fitted a (binomial/logistic) GLMM to the data (in statsmodels, but here it is in lme4 syntax):

success ~ C(cohort) + C(trial) + age + difficulty + C(gender) + (1|participant)

where success is a binary (yes|no), cohort is 1 or 2, trial is 1 for the first trial (cohort 1 in $C_A$, cohort 2 in $C_B$) or two for the switched conditions, there are several parameters (like age, gender, size) related to the participant, several parameters related to the sample (difficulty, temperature, etc) and I include a random offset per participants.

This works fine and tells me which parameter has an effect on the success.

But what I am interested in is what changed between between $C_A$ of the the first cohort vs. $C_A$ of the second cohort. Ideally with the additional information of "what changed between cohort 1 and 2 in $C_A$ that has not changed between cohort 1 and 2 in $C_B$.

Is there a way to express this question in a GLMM?

edit (answering @Erik):

There are basically 4 conditions, cohort 1 and 2 who both did both tasks $C_A$ and $C_B$ but with switched task order. Expressed as a (reduced) table:

cohort , trial , condition , participant , age , [...] , success
1      , 1     , A         , 0           , 20  , [...] , 1
1      , 2     , B         , 0           , 20  , [...] , 0
1      , 1     , A         , 1           , 21  , [...] , 1
1      , 2     , B         , 1           , 21  , [...] , 0
[...]  , [...] , [...]     , [...]       , []  , [...] , [...]
2      , 1     , B         , n-1         , 40  , [...] , 1
2      , 2     , A         , n-1         , 40  , [...] , 0
2      , 1     , B         , n           , 41  , [...] , 1
2      , 2     , A         , n1          , 41  , [...] , 0

Notice that trial and condition are switched for the second cohort. Repetitions per participant are left out for brevity.

edit 2 (answering to @Erik again)

I am not exactly interested in the question "did a factor have an effect on the result" (eg: "did age have an effect on success"). That would be success ~ age + (1|participant)

I am interested in the question "what factor can explain a difference in success between the two conditions in the first trial". [1]

What I hope for is that the output will tell me (exemplary): "age does not explain the difference between the two cohorts, but difficulty had a more drastic impact on success in cohort 1 than it had in cohort 2".

(I'll note that I don't know if something like this is even possible with GLMMs, I have just recently read up on them from what is available on the Internet. If there are any "must read" articles or books on the topic I would be interested too.)

[1] I'll leave the additional question about the second trial out for now.

Answer 1

I believe that part of what you want to do is test the interaction between cohort and condition. This is most obvious in your statement about wanting to know about:

what changed between between $C_A$ of the the first cohort vs. $C_A$ of the second cohort

This can be easily determined through the following model, assuming that you code $C_A$ as 0, $C_B$ as 1, cohort 1 as 0, and cohort 2 as 1:

m1 <- success ~ cohort*condition + trial + age + difficulty + gender + (1|participant)

The coefficient for cohort indicates the difference in log odds between cohort 1 and 2 at condition==0, which if you've coded condition as suggested would be $C_A$.

Then you mention:

Ideally with the additional information of "what changed between cohort 1 and 2 in $C_A$ that has not changed between cohort 1 and 2 in $C_B$.

Going through the logic of the coefficients for the interaction, remember that the (Intercept) is giving you the log odds for individuals with 0 on condition and cohort, so folks in cohort 1 and $C_A$.

The coefficient for condition indicates the difference in log odds between $C_A$ and $C_B$ for cohort==0, which again, if you've recoded as suggested this would be cohort 1.

Then you have the coefficient of the interaction term, in R this will print as cohort:condition. This tells you the log odds for individuals having the 1 values for cohort and condition, so folks who were in cohort 2 and $C_B$. You can also interpret this as the difference in log odds for $C_B$ associated with going from cohort 1 to 2.

Thus, in order to test whether there was a difference in the contrast between cohort 1 and 2 in $C_A$ than the contrast between cohort 1 and 2 in $C_B$, you would need to run a post-estimation test of the equality of the coefficients for cohort and cohort:condition. You can use glht() in the multcomp() package in R for this or possibly emmeans(). I'm not sure how this would be implemented in statsmodels.

Edit in response to updated question

I am interested in the question "what factor can explain a difference in success between the two conditions in the first trial".

To answer this question for age, for example, you need to add a further three-way interaction between condition, trial, and age, which I'll write out all the terms this will expand to:

m2 <- success ~ cohort + condition + trial + age + condition:trial + condition:age + trial:age + condition:trial:age + difficulty + gender + (1|participant)

Three-way interactions are somewhat challenging to interpret. Remember that the main effects of interaction terms are interpreted as the coefficient for that variable when the variable it is interacted with is 0. Explaining each of the interaction terms would take a lot of text. Instead of me doing that I suggest that you use a package such as effects() or ggeffects() in R to get the marginal means for various permutations of these variables and plot them.