I am interested in how to demonstrate that two groups are equivalent in their odds of making a correct response at post-intervention, controlling for their repsonse pre-intervention.
In my experiment there are two groups of participants who receive the same intervention, teaching them how to play a trump card correctly (binary variable, 0 for incorrect, 1 for correct). They are tested before the intervention to see if they play the trump correctly, and then again after the intervention (binary variable
baseline = 0,
followup = 1). Before they sign up for the study researchers ask them if they have played any trump-style card game before they enrolled in the study (binary variable
not trained = 0 vs
trained = 1). Researchers think that the people who have played a trump-style game before (the
trained group) will have a high likelihood of playing the trump correctly at both
followup, but that the people who have never played a trump-style game (
not trained group) will have a lower likelihood of playing the trump correctly at baseline than the trained group. The researchers also hypothesise that the
not trained group's likelihood of playing the trump correctly after the intervention , at followup, will be equivalent to the
trained group's likelihood at followup. This study is based on the theory that once you have learned something simple, extra training will not make you learn it any better.
Here is the data
library(GLMMadaptive) lbrary(dplyr) d <- data.frame(id = 1:94, training = c(0,1,1,0,0,0,0,1,0,1,1,1,0,1,0,0,1,1,0,0,1,1,0,1,0,0,0,0,0,1,0,0,0,0,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,1,0,1,1,1,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,1,1,0,1,0,1,0,0,1,0,0,0,0,1,0,1), baseline = c(1,1,1,1,1,0,1,1,1,1,0,1,0,1,1,1,1,1,0,1,1,1,0,1,0,1,0,0,0,1,1,1,1,0,0,1,1,1,1,0,1,1,0,1,0,1,1,1,0,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,0,1,1,0,1,0,1,0,1,0,1,0,1,0,0,1,0,1,0,1,0,1,1,1,1,1,1,1,1,0), followup = c(1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1))
Let's look at the proportions correct in each cell - baseline/not trained, baseline/trained, followup/not trained, followup/trained .
d %>% group_by(training) %>% summarise(countBase = sum(baseline), countFU = sum(followup), tot = n()) %>% mutate(percBase = round(100*countBase/tot,2), percFU = round(100*countFU/tot,2)) %>% transform(training = ifelse(training == 0, "not trained", "trained")) # output # training countBase countFU tot percBase percFU # 1 not trained 40 62 65 61.54 95.38 # 2 trained 26 28 29 89.66 96.55
At face value it looks like the theory holds. The
not trained group, who were not trained in how to play trump-style card games (n= 65) had a lower proportion of people who played the trump correctly before the intervention (40/65 = 61.5%) than people who had played trump-style card games before (n = 29, 26/29 = 89.7%). But, after being trained during the intervention in how to play trump-style card card games, the proportion who played the trump correctly in the
not trained group increased to 62/65 = 95.4%. The trained group really had nowhere to go, thus the proportion who played the trump correctly post-intervention increased on slightly to 96.6% (28/29).
Now because this is repeated measures, it's not enough to look at the proportions of correct responses in each cell, we also have to look at what percentages of each group changed
d$changed <- ifelse(d$baseline == 0 & d$followup == 0, "bothCorrect", ifelse(d$baseline == 0 & d$followup == 1, "incorrectToCorrect", ifelse(d$baseline == 1 & d$followup == 0, "correctToIncorrect", "bothCorrect"))) changeDF <- data.frame(table(d$training, d$changed)) %>% dplyr::rename(training = Var1, changed = Var2) %>% arrange(training) %>% group_by(training) %>% mutate(tot = sum(Freq), perc = round(100*Freq/tot,2))) # output # training changed Freq tot perc # 1 0 bothCorrect 41 65 63.1 # 2 0 correctToIncorrect 1 65 1.54 # 3 0 incorrectToCorrect 23 65 35.4 # 4 1 bothCorrect 25 29 86.2 # 5 1 correctToIncorrect 1 29 3.45 # 6 1 incorrectToCorrect 3 29 10.3
Clearly many more people changed in the group who had received no training prior to the intervention, mostly people changing from playing the trump incorrectly to playing it correctly. Very few people who had received prior training changed (basically because most of them already endorsed the view at baseline)
Now for the longitudinal logistic regression. I have used the
mixed_model function from the
GLMMadaptive package. This is very similar to the
glmer() function from
lme4 but it allows for the calculation of marginal coefficients (rather than the subject specific coefficients, see here)
# convert wide dataset to long longDF <- d %>% dplyr::select(-one_of("changed")) %>% gather(key = time, value = correct, baseline, followup) %>% mutate(id = factor(id), training = factor(ifelse(training == 0, "not trained", "trained"), levels = c("not trained", "trained")), correct = factor(ifelse(correct == 0, "incorrect", "correct"), levels = c("incorrect", "correct")), time = factor(time)) # now for the glmm mod <- mixed_model(correct ~ training*time, data = longDF, random = ~ 1|id, family = binomial()) # get marginal coefficients (this takes about 30 seconds to run) margMod <- marginal_coefs(mod, std_errors = T, cores = 3) margMod # output # Estimate Std.Err z-value p-value # (Intercept) 0.4726 0.2701 1.7495 0.08020486 # trainingtrained 1.6864 0.6563 2.5694 0.01018619 # timefollowup 2.5595 0.7607 3.3648 0.00076599 # trainingtrained:timefollowup -1.3892 1.2549 -1.1070 0.26829049
Now let's calculate odds ratios and CIs for the difference in odds of being correct due to
baseline and at
orAtBase <- exp(c(est = margMod$coef_table[2,1], lowCI = margMod$coef_table[2,1] - 1.96*margMod$coef_table[2,2], hiCI = margMod$coef_table[2,1] + 1.96*margMod$coef_table[2,2])) orAtFU <- exp(c(est = margMod$coef_table[4,1], lowCI = margMod$coef_table[4,1] - 1.96*margMod$coef_table[4,2], hiCI = margMod$coef_table[4,1] + 1.96*margMod$coef_table[4,2])) rbind(orAtBase, orAtFU) # output # est lowCI hiCI # orAtBase 5.400231 1.49183286 19.548101 # orAtFU 0.249287 0.02130753 2.916528
This is how I interpret these results:
- The odds of playing the trump correctly at baseline were 5.4 times higher (95%CI: 1.5, 19.5) if participants had had prior experience with trump-style card games than if they had no experience with such games.
- The odds of playing the card correctly after the intervention controlling for how they played the card at baseline were 1/0.249287 = 4 times higher in the group who had not received training than in the group who had, but, given the extremely wide confidence intervals that overlap 1, it is not possible to say this difference is due to anything more than chance.
The above analysis and interpretation seem ok, but they don't really get at what I want to get at, which is that the two groups seem to have roughly the same proportion at
followup. How would I go about demonstrating this equivalence? Is there some way to conduct simple effects of training at post-follow-up only using the coefficients from the regression output and computing the standard error manually (via methods explained in the answers here)? If so which coefficients do I add? And, if it is not possible to use the existing regression coefficients would some kind of Bayes Factor or non-inferiority test be better?
My interpretation of the model doesn't really refer to the fact that the
no traininggroup increased their knowledge of how to play the trump, whereas as the
traininggroup did not, because the
traininggroup mostly already knew before the intervention. Would it also be correct to say "the increase in odds of playing the trump correctly from baseline to followup was 4 times higher in the non-training group than the training group"?
The odds of the
non trainedgroup playing the trump correctly at followup were exp(2.5595) = 12.9 times higher than they were at baseline. So how do I perform the same calculation in the