Compare the evolution of a binary variable between two groups of a panel

Question

I would like to compare the evolution of scores on a math test between two groups in a longitudinal manner. I have a panel of students that we tested in June, at the end of the year, and then in September, at the beginning of the school year. I am trying to find out if the vacations have a differential effect according to the gender of the student. To do this, I compare the proportion of men who obtained a score defined as "satisfying", to the proportion of women who obtained a satisfying score, in June then in September. How do I know if the evolution of the gap between men and women is significant?

Let me give you a concrete example to make it more clear:

• Let's say that in June, 76% of men score well and 81% of women score well. So the gap between men and women is 5 points.

• In September, 71% of men score well and 79% of women score well. The gap between men and women is therefore 8 points this time.

How can we test if the school vacations had a different effect by gender? What type of test should be used to determine whether the change from a 5-point gap between men and women in June to an 8-point gap in September is a significant change?

I thought about a Two-Sample McNemar Test. Unfortunately I didn't find any implementation of this test in R.

Answer 1

Since the response variable is binary (Well / Not-well), you can use logistic regression.

You will need to know the count for each cell (Men-June-Well, Men-June-NotWell, and so on).

If you have the same individuals across time, it is helpful to be able to identify each individual across time. (You would need this information for McNemar test also). If so, you can use mixed-effects logistic regression.

I think it is the Gender x Time interaction you are interested in.

The following example is just a logistic regression model without repeated measures, analogous to a two-way factorial anova.

Data = read.table(header=TRUE, stringsAsFactors=TRUE, text="

Time  Gender  Result   Count
June  Men     Well     76
June  Men     NotWell  24
June  Women   Well     81
June  Women   NotWell  19
Sept  Men     Well     71
Sept  Men     NotWell  29
Sept  Women   Well     79
Sept  Women   NotWell  21
")

library(tidyr)
Long = uncount(Data, Count)
rownames(Long) = seq(1:nrow(Long))

XT = xtabs(~ Time + Gender + Result, data=Long)

XT

model = glm(Result ~ Gender + Time + Gender:Time, data=Long, family=binomial())

library(car)

Anova(model)

   ### Analysis of Deviance Table (Type II tests)
   ### 
   ###             LR Chisq Df Pr(>Chisq)
   ### Gender       2.37792  1     0.1231
   ### Time         0.69110  1     0.4058
   ### Gender:Time  0.07645  1     0.7822

library(emmeans)

marginal = emmeans(model, ~ Gender:Time, type="response")

marginal

   ###  Gender Time prob     SE  df asymp.LCL asymp.UCL
   ###  Men    June 0.76 0.0427 Inf     0.667     0.834
   ###  Women  June 0.81 0.0392 Inf     0.721     0.875
   ###  Men    Sept 0.71 0.0454 Inf     0.614     0.790
   ###  Women  Sept 0.79 0.0407 Inf     0.699     0.859
   ### 
   ### Confidence level used: 0.95 
   ### Intervals are back-transformed from the logit scale 

pairs(marginal)

   ###  contrast                odds.ratio    SE  df null z.ratio p.value
   ###  Men June / Women June        0.743 0.257 Inf    1  -0.859  0.8259
   ###  Men June / Men Sept          1.293 0.416 Inf    1   0.800  0.8544
   ###  Men June / Women Sept        0.842 0.286 Inf    1  -0.508  0.9573
   ###  Women June / Men Sept        1.741 0.587 Inf    1   1.646  0.3528
   ###  Women June / Women Sept      1.133 0.401 Inf    1   0.353  0.9849
   ###  Men Sept / Women Sept        0.651 0.215 Inf    1  -1.302  0.5616
   ### 
   ### P value adjustment: tukey method for comparing a family of 4 estimates 
   ### Tests are performed on the log odds ratio scale