Background. I'm analyzing observational data where the "treatment" is a supplemental program to improve student outcomes in several university science and math courses. I have data on a few thousand students collected over several semesters and several courses. Students self-select into the treatment. (I plan to address this with propensity score matching, but I'm starting here with just a regression model in order to focus on the questions below without addressing PS matching.)
I also have data on a range of covariates, including student demographics, SAT scores, high school GPA, university GPA at the start of the course, major, etc. The outcome is grade in the course (I'm also looking at a logistic regression of student retention and graduation in subsequent semesters), which I'm treating as a continuous variable ranging from 0 ("F") to 4 ("A") (though I realize it might make sense to treat this as an ordinal categorical outcome).
Here are my questions:
1. Dealing with multiple courses. I want to estimate the effect of the treatment on course grade, but this might vary by course. Although the supplemental program is run centrally, the courses span chemistry, math and biology, different people deliver the supplemental program for each course, and they are trained separately for each course. Does it make more sense to deal with this by modeling each course separately or with a single model with an interaction between treatment and course?
2. Repeated measures: Course repeats. The data includes one row for each student for each semester the student was enrolled in a given course. Thus, the same student can have multiple observations if they took a given course more than once, for example, because they failed the first time. I could run a regression such as (in R notation)
lm(grade ~ treatment + course + repeating + sat.math + sat.verbal + high.school.gpa + gpa.at.start.of.course + [additional covariates]) where:
treatmentis a dummy for whether the student received the treatment (and, depending on the answer to the first question, the model could also be
treatment + course).
courseis a factor, marking which of the six possible courses the grade applies to.
repeatingis a dummy for whether the student took the same course at least once before.
Does conditioning on
repeating account for the within-subject repeated measures correlation (in effect "de-clustering" the data), or would it be better/more valid to instead use a hierarchical model that accounts for the repeated measures of student within course? Note also that although most covariates (e.g., demographics, SAT scores) don't change, a few other covariates (e.g., university GPA, total units earned), vary by semester. Once again, does conditioning on these varying factors "de-cluster" the data, or is a repeated measures framework more appropriate?
3. Multiple treatment exposures. At the time of a given observation, a student might have previously received the treatment in the same or a different course and/or might be taking two courses during the same semester and receiving the treatment in both courses. I could condition on additional covariates in the linear regression, such as
previousTreatment, equal to the number of times the student received the treatment in previous semesters, and
currentTreatment as a dummy marking whether the student is receiving the treatment in another course during the same semester. Is this the way to control for previously or concomitantly receiving the treatment, or is a different approach warranted, given that this is once again a case of repeated measures for some students?
In addition to answers explaining the conceptual issues, to make this more concrete I'd also be interested in R code snippets that illustrate suggested model structures.