# How to model a complex treatment and hierarchical structure in regression with observational data

Background. I'm analyzing observational data where the "treatment" is a supplemental program to improve student outcomes in several university science and math courses. I have data on a few thousand students collected over several semesters and several courses. Students self-select into the treatment. (I plan to address this with propensity score matching, but I'm starting here with just a regression model in order to focus on the questions below without addressing PS matching.)

I also have data on a range of covariates, including student demographics, SAT scores, high school GPA, university GPA at the start of the course, major, etc. The outcome is grade in the course (I'm also looking at a logistic regression of student retention and graduation in subsequent semesters), which I'm treating as a continuous variable ranging from 0 ("F") to 4 ("A") (though I realize it might make sense to treat this as an ordinal categorical outcome).

Here are my questions:

1. Dealing with multiple courses. I want to estimate the effect of the treatment on course grade, but this might vary by course. Although the supplemental program is run centrally, the courses span chemistry, math and biology, different people deliver the supplemental program for each course, and they are trained separately for each course. Does it make more sense to deal with this by modeling each course separately or with a single model with an interaction between treatment and course?

2. Repeated measures: Course repeats. The data includes one row for each student for each semester the student was enrolled in a given course. Thus, the same student can have multiple observations if they took a given course more than once, for example, because they failed the first time. I could run a regression such as (in R notation) lm(grade ~ treatment + course + repeating + sat.math + sat.verbal + high.school.gpa + gpa.at.start.of.course + [additional covariates]) where:

• treatment is a dummy for whether the student received the treatment (and, depending on the answer to the first question, the model could also be treatment*course instead of treatment + course).
• course is a factor, marking which of the six possible courses the grade applies to.
• repeating is a dummy for whether the student took the same course at least once before.

Does conditioning on repeating account for the within-subject repeated measures correlation (in effect "de-clustering" the data), or would it be better/more valid to instead use a hierarchical model that accounts for the repeated measures of student within course? Note also that although most covariates (e.g., demographics, SAT scores) don't change, a few other covariates (e.g., university GPA, total units earned), vary by semester. Once again, does conditioning on these varying factors "de-cluster" the data, or is a repeated measures framework more appropriate?

3. Multiple treatment exposures. At the time of a given observation, a student might have previously received the treatment in the same or a different course and/or might be taking two courses during the same semester and receiving the treatment in both courses. I could condition on additional covariates in the linear regression, such as previousTreatment, equal to the number of times the student received the treatment in previous semesters, and currentTreatment as a dummy marking whether the student is receiving the treatment in another course during the same semester. Is this the way to control for previously or concomitantly receiving the treatment, or is a different approach warranted, given that this is once again a case of repeated measures for some students?

In addition to answers explaining the conceptual issues, to make this more concrete I'd also be interested in R code snippets that illustrate suggested model structures.

• Good, solid methodological questions. Of course you're familiar with the literature on hierarchical modeling including Gelman's Data Analysis Using Regression and Multilevel/Hierachical Models for a Bayesian treatment and Singer's Longitudinal Data Analysis for a more traditional approach. To me, Singer's book has more nitty-gritty, hands-on advice. This paper by Russ Wolfinger is more widely generalizable beyond the SAS software and does a good job of comparing trade-offs between repeated measures and hierarchical mixed models. ats.ucla.edu/stat/sas/library/mixedglm.pdf Oct 22, 2015 at 14:47
• I have both of those books on my desk :). Thanks for the pointer to the Wolfinger article. Oct 22, 2015 at 15:07

## 1 Answer

Q1. The most concise approach is to use an interaction term. However, with enough complexity in what you introduce and analyze, you may find it easier to keep track of results if you build separate models by course. Also, if you want to gauge the effect of outliers, or study residuals, the second approach will give you more flexibility to assess.

Q2. I'm not sure your use of "conditioning on..." is correct. In any case, including "repeating" as a predictor will not solve the problem of correlated observations. Your project is already ambitious, dependent on many assumptions, and thus fraught with a certain amount of risk for accuracy, validity, interpretability, and acceptance by your intended audience. Thus my suggestion is that in this respect you simplify by including only the first semester for any student.

Q3. I admire your courage and resourcefulness in tackling these questions, but again I'd recommend excluding any courses beyond the first for any student. You won't be losing much solid explanatory information: consider the many factors that, aside from the treatment, could be accounting for outcomes at stages beyond the first semester (e.g., see An inventory: 11 issues with value-added studies ).