I am interested in how to demonstrate that two groups are equivalent in their odds of making a correct response at post-intervention, controlling for their repsonse pre-intervention.


In my experiment there are two groups of participants who receive the same intervention, teaching them how to play a trump card correctly (binary variable, 0 for incorrect, 1 for correct). They are tested before the intervention to see if they play the trump correctly, and then again after the intervention (binary variable time: baseline = 0, followup = 1). Before they sign up for the study researchers ask them if they have played any trump-style card game before they enrolled in the study (binary variable training: not trained = 0 vs trained = 1). Researchers think that the people who have played a trump-style game before (the trained group) will have a high likelihood of playing the trump correctly at both baseline and followup, but that the people who have never played a trump-style game (not trained group) will have a lower likelihood of playing the trump correctly at baseline than the trained group. The researchers also hypothesise that the not trained group's likelihood of playing the trump correctly after the intervention , at followup, will be equivalent to the trained group's likelihood at followup. This study is based on the theory that once you have learned something simple, extra training will not make you learn it any better.


Here is the data


d <- data.frame(id = 1:94,
                training = c(0,1,1,0,0,0,0,1,0,1,1,1,0,1,0,0,1,1,0,0,1,1,0,1,0,0,0,0,0,1,0,0,0,0,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,1,0,1,1,1,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,1,1,0,1,0,1,0,0,1,0,0,0,0,1,0,1),
                baseline = c(1,1,1,1,1,0,1,1,1,1,0,1,0,1,1,1,1,1,0,1,1,1,0,1,0,1,0,0,0,1,1,1,1,0,0,1,1,1,1,0,1,1,0,1,0,1,1,1,0,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,0,1,1,0,1,0,1,0,1,0,1,0,1,0,0,1,0,1,0,1,0,1,1,1,1,1,1,1,1,0),
                followup = c(1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1))

Let's look at the proportions correct in each cell - baseline/not trained, baseline/trained, followup/not trained, followup/trained .

d %>% group_by(training) %>%
      summarise(countBase = sum(baseline),
                countFU = sum(followup),
                tot = n()) %>%
      mutate(percBase = round(100*countBase/tot,2), percFU = round(100*countFU/tot,2)) %>%
      transform(training = ifelse(training == 0, "not trained", "trained"))

#    output
#         training countBase countFU tot percBase percFU
#    1 not trained        40      62  65    61.54  95.38
#    2     trained        26      28  29    89.66  96.55

At face value it looks like the theory holds. The not trained group, who were not trained in how to play trump-style card games (n= 65) had a lower proportion of people who played the trump correctly before the intervention (40/65 = 61.5%) than people who had played trump-style card games before (n = 29, 26/29 = 89.7%). But, after being trained during the intervention in how to play trump-style card card games, the proportion who played the trump correctly in the not trained group increased to 62/65 = 95.4%. The trained group really had nowhere to go, thus the proportion who played the trump correctly post-intervention increased on slightly to 96.6% (28/29).

Now because this is repeated measures, it's not enough to look at the proportions of correct responses in each cell, we also have to look at what percentages of each group changed

d$changed <- ifelse(d$baseline == 0 & d$followup == 0, "bothCorrect",
                  ifelse(d$baseline == 0 & d$followup == 1, "incorrectToCorrect",
                         ifelse(d$baseline == 1 & d$followup == 0, "correctToIncorrect", "bothCorrect")))

changeDF <- data.frame(table(d$training, d$changed)) %>%
            dplyr::rename(training = Var1, changed = Var2) %>%
            arrange(training) %>%
            group_by(training) %>%
            mutate(tot = sum(Freq),
                   perc = round(100*Freq/tot,2)))

#  output
#   training changed             Freq   tot  perc
# 1 0        bothCorrect           41    65 63.1 
# 2 0        correctToIncorrect     1    65  1.54
# 3 0        incorrectToCorrect    23    65 35.4 
# 4 1        bothCorrect           25    29 86.2 
# 5 1        correctToIncorrect     1    29  3.45
# 6 1        incorrectToCorrect     3    29 10.3 

Clearly many more people changed in the group who had received no training prior to the intervention, mostly people changing from playing the trump incorrectly to playing it correctly. Very few people who had received prior training changed (basically because most of them already endorsed the view at baseline)

Inferential Analysis

Now for the longitudinal logistic regression. I have used the mixed_model function from the GLMMadaptive package. This is very similar to the glmer() function from lme4 but it allows for the calculation of marginal coefficients (rather than the subject specific coefficients, see here)

# convert wide dataset to long
longDF <- d %>% dplyr::select(-one_of("changed")) %>% 
                gather(key = time, value = correct, baseline, followup) %>%
                mutate(id = factor(id),
                       training = factor(ifelse(training == 0, "not trained", "trained"), levels = c("not trained", "trained")),
                       correct = factor(ifelse(correct == 0, "incorrect", "correct"), levels = c("incorrect", "correct")),
                       time = factor(time))

# now for the glmm
mod <- mixed_model(correct ~ training*time,
                   data = longDF,
                   random = ~ 1|id,
                   family = binomial())

# get marginal coefficients (this takes about 30 seconds to run)
margMod <- marginal_coefs(mod, std_errors = T, cores = 3)

# output
#                              Estimate Std.Err z-value    p-value
# (Intercept)                    0.4726  0.2701  1.7495 0.08020486
# trainingtrained                1.6864  0.6563  2.5694 0.01018619
# timefollowup                   2.5595  0.7607  3.3648 0.00076599
# trainingtrained:timefollowup  -1.3892  1.2549 -1.1070 0.26829049

Now let's calculate odds ratios and CIs for the difference in odds of being correct due to training at baseline and at followup.

orAtBase <- exp(c(est = margMod$coef_table[2,1], 
                    lowCI = margMod$coef_table[2,1] - 1.96*margMod$coef_table[2,2],
                    hiCI = margMod$coef_table[2,1] + 1.96*margMod$coef_table[2,2]))

orAtFU <- exp(c(est = margMod$coef_table[4,1], 
                lowCI = margMod$coef_table[4,1] - 1.96*margMod$coef_table[4,2],
                hiCI = margMod$coef_table[4,1] + 1.96*margMod$coef_table[4,2]))

rbind(orAtBase, orAtFU)

# output
#               est      lowCI      hiCI
# orAtBase 5.400231 1.49183286 19.548101
# orAtFU   0.249287 0.02130753  2.916528


This is how I interpret these results:

  1. The odds of playing the trump correctly at baseline were 5.4 times higher (95%CI: 1.5, 19.5) if participants had had prior experience with trump-style card games than if they had no experience with such games.
  2. The odds of playing the card correctly after the intervention controlling for how they played the card at baseline were 1/0.249287 = 4 times higher in the group who had not received training than in the group who had, but, given the extremely wide confidence intervals that overlap 1, it is not possible to say this difference is due to anything more than chance.


  1. The above analysis and interpretation seem ok, but they don't really get at what I want to get at, which is that the two groups seem to have roughly the same proportion at followup. How would I go about demonstrating this equivalence? Is there some way to conduct simple effects of training at post-follow-up only using the coefficients from the regression output and computing the standard error manually (via methods explained in the answers here)? If so which coefficients do I add? And, if it is not possible to use the existing regression coefficients would some kind of Bayes Factor or non-inferiority test be better?

  2. My interpretation of the model doesn't really refer to the fact that the no training group increased their knowledge of how to play the trump, whereas as the training group did not, because the training group mostly already knew before the intervention. Would it also be correct to say "the increase in odds of playing the trump correctly from baseline to followup was 4 times higher in the non-training group than the training group"?

  3. The odds of the non trained group playing the trump correctly at followup were exp(2.5595) = 12.9 times higher than they were at baseline. So how do I perform the same calculation in the trained group only?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.