0
$\begingroup$

I've implemented a learning technology to reduce the gender gap(the difference between male and female student grades) in CS education. I conducted a quasi-experiment across six semesters, with ~200 students and similar numbers of males and females in each semester. The first three semesters were used as the control and we implemented the intervention in the second three semesters. To estimate the effect of the intervention on gender gap, I used the following model:

Model1 <- betareg(CourseGrade ~ Treatment * Gender)

Where each row in the dataset indicates a single student and Treatment is a dummy that indicates whether the observation was in semesters before or after the implementation of the intervention. However, because the students and exams were different semester to semester, the gender gap is different across semesters. To control for the semester-specific effect, I implemented the following model:

Model2 <- betareg(CourseGrade ~ Semester * Gender)
Grid2 <- ref_grid( Model2 )
Treatment_Fact <- factor(c("Control", "Control", "Control", "Treated", "Treated", "Treated"), levels = c("Control", "Treated"))
Grid2 <- add_grouping(Grid2, "Condition", "Semester", Treatment_Fact)
emmeans(Grid2, revpairwise ~ Gender | Condition, type = "response")

However, it prints out:

Results are averaged over the levels of: Semester

I don't understand why! According to the documentation, the reference factor (Semester) should be nested in the grouping factor (Condition). If it correctly reports the aggregate effect, the results should not be averaged over the levels of the Semester.

Is there anything wrong with my analysis or interpretation? Is there any more reasonable way of estimating the effect size?

$\endgroup$
2
  • $\begingroup$ Professor Russell V. Lenth just responded to me outside StackOverflow. I'm sharing his response here because I found it quite helpful: Semester remains a factor in the grid -- Condition is a new factor that is added to the grid. So when marginalizing to just Gender and Condition, we average over Semester, separately in each Condition. It may help understand this if you look at summary(Grid2, by = "Gender") and emmeans(Grid2, ~ Condition | Gender) and note that the Condition*Gender means are averaged over Semester separately within each Condition. $\endgroup$ Commented Dec 5, 2019 at 21:53
  • $\begingroup$ He also added: In other words, each result you see is averaged over three of the six semesters -- not all six semesters. Which three semesters are averaged over depends on Condition. $\endgroup$ Commented Dec 5, 2019 at 21:57

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.