I've implemented a learning technology to reduce the gender gap(the difference between male and female student grades) in CS education. I conducted a quasi-experiment across six semesters, with ~200 students and similar numbers of males and females in each semester. The first three semesters were used as the control and we implemented the intervention in the second three semesters. To estimate the effect of the intervention on gender gap, I used the following model:
Model1 <- betareg(CourseGrade ~ Treatment * Gender)
Where each row in the dataset indicates a single student and Treatment is a dummy that indicates whether the observation was in semesters before or after the implementation of the intervention. However, because the students and exams were different semester to semester, the gender gap is different across semesters. To control for the semester-specific effect, I implemented the following model:
Model2 <- betareg(CourseGrade ~ Semester * Gender)
Grid2 <- ref_grid( Model2 )
Treatment_Fact <- factor(c("Control", "Control", "Control", "Treated", "Treated", "Treated"), levels = c("Control", "Treated"))
Grid2 <- add_grouping(Grid2, "Condition", "Semester", Treatment_Fact)
emmeans(Grid2, revpairwise ~ Gender | Condition, type = "response")
However, it prints out:
Results are averaged over the levels of: Semester
I don't understand why! According to the documentation, the reference factor (Semester) should be nested in the grouping factor (Condition). If it correctly reports the aggregate effect, the results should not be averaged over the levels of the Semester.
Is there anything wrong with my analysis or interpretation? Is there any more reasonable way of estimating the effect size?
summary(Grid2, by = "Gender")
andemmeans(Grid2, ~ Condition | Gender)
and note that the Condition*Gender means are averaged over Semester separately within each Condition. $\endgroup$