Mixed Effects Logistic Regression w/ Repeated Measures for Observational Data

Question

I am working on a study of how different scholarship programs at a university may influence student retention (i.e., if students are still enrolled at the university one year later). Each student can receive a different combination of scholarships--they can also lose or gain scholarships over time. All students in the study started in the same cohort/semester.

My data has the following variables:

• Student ID: identifier for each student
• Term: semester that the student had the scholarship (categorical)
• Award Type: whether scholarships are awarded based on merit, financial need, or special interest (categorical)
• Award Prop: the proportion of student expenses that the scholarship covers - for example, an Award Prop of 0.50 means that the scholarship will cover half of a student's college expenses (continuous)
• Outcome: whether or not the student persisted a year later (binary)

An example of the data is provided below:

ID     Term           Award Type   Award Prop   Outcome (Persisted?)
0025   Fall 2012      Merit        0.25         Y
0025   Fall 2012      Need         0.50         Y
0025   Spring 2013    Need         0.50         Y
1310   Fall 2012      Interest     0.30         N
1310   Spring 2013    Interest     0.30         N
1229   Spring 2013    Need         0.90         Y
1229   Spring 2013    Interest     0.10         Y
2843   Fall 2012      Merit        0.50         N

Due to the fact that I'm modeling a binary outcome for individuals sampled several times, I am thinking of approaching this with a mixed effects logistic regression with repeated measures using lme4 in R.

I think the way to structure this might be to nest ID within Term as a random effect:

glmer(Outcome ~ AwardType + AwardProp + 1|(Term/ID), family=binomial)

First, based on the parameters above, does this seem to be a sound way to approach this analysis?

Second, should I be concerned about potential issues with missing/unbalanced data (since each person has different scholarships, a person with a scholarship one semester may not have it in other semesters, etc.)? One article I read said that mixed effects modeling should take care of this.

Answer 1

Have you thought about using a survival model (cox regression) here? I think such a model will take the temporal characteristic of your data better into account. Given that your treatment/exposure can vary over time, it also make sense to look at models for time varying exposures, e.g. here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4351798/

Answer 2

Your analysis might be okay for a randomized trial with one observation period. What is problematic with observational data includes:

• You do not model the "treatment" (scholarship) assignment process. If some scholarships are given to, say, underprivileged students who may be more likely to drop-out in the first place, while another scholarship is given to e.g. athletes or legacy admissions or highly performing students that are more likely to stay in the course in the first place, then you end up with confounding. You can't tell the difference between whether you are comparing the students as they were (i.e. the same would have happened even if they had not got the scholarship), the causal effect of the scholarship, or (most likely) a mixture of the two (but likely with a strong effect of the pre-existing student characteristics).
  • To account for this, you'd need to have model the assignment process, which likely requires quite an in-depth level of information (e.g. details on socio-economic background, financial situation, education of parents/other key family members, past academic track-record etc.). Without those things, it'll be futile to hope to account for the possible sources of confounding and you're likely just wasting your time.
  • If you have those things, you can try various causal inference methods (e.g. adjustment, doubly-robust methods, propensity score methods etc.). One good book on the topic would be this one, a pre-print is freely accessible at the link location on the author's webpage.
• This only gets worse, if you treat time periods the way you do. E.g. if a student has a scholarship for a period, then looses their scholarship, then has a period without it and then drops-out, causally attributing this to not having a scholarship is really problematic. The reason is that perhaps the reason that explains them dropping-out is what made then loose the scholarship (e.g. dropping test performance) or it is a direct effect of the loss of the scholarship (or some mixture of these things).
• I'm not sure whether you have students without any scholarships at any time in your data, but without those it's also a lot harder to say things about the causal effects of scholarships (you can at best but even that gets a lot harder compare types of scholarships). It may be cleaner to compare those having a scholarship at a fixed point of time (even if they may loose it in the future) going forwards with time-to-event methods.

Answer 3

It seems like you may be getting a lot of overlaps in the 1-year follow-up periods for different scholarships obtained by the same student (e.g., first and second observations in your example). I don't know if it is true in your case, but I suppose it is also possible for a student to obtain a new scholarship during the 1-year follow-up period from a previous scholarship. So rather than simply having correlated observations for the same individual, you'll have different observations for the same outcome. My suggestion would be that rather than treating scholarships as separate observations, you could combine current observations, such that you have a clean 1-year follow-up period per observation (clean meaning no additional scholarships granted and no scholarships lost). Then the predictor variables can be constructed to indicate what scholarships were obtained leading up to the start of that 1-year period. The outcome variable would still measure if the student was still enrolled at the university one year later.