Comparing the effect of a treatment that was optional for its receivers

Question

In 2012 we collected data at our university about retention of students from first semester to second semester along with some other variables. The retention variable is binary, 'retained' or 'did not retain'.

In 2013 we introduced a new system. Student success advisers were assigned and any student when needed could go to them. We collected some new variables associated with the introduction of the student success advisers together with those collected in 2012. Now, going to these advisers was not mandatory. Those who felt a problem could go to them.

Just looking at 2013 data alone makes it seem that the advisers make things worse: the students they see do worse, and are more likely to leave university. Only the students who faced problems probably went to the advisers and those who were doing well probably didn't. If 10 students went to the advisers with problems and 8 retained to semester 2, that is surely a success. But if 80 out of 100 students who didn't have any problem and thus didn't go to the adviser retained to semester 2, that will have log odds similar to those who went to the advisers and thus tell you that going to the advisers had no significant effect!

Part of the reason was a major policy change that affected how universities in our country took in students. There was a "cap" in place before 2013, a limit on how many students a university could take on and expect government funding for. In 2013 that was removed. So, successful, desirable universities started opening their doors to more students, and more students were able, suddenly, to get into desirable universities. Our university is not so desirable. So what probably happened was that our university was forced in 2013 to take significantly "worse" students, which could well lead to greater retention difficulty.

What my bosses think is that we need to weight the 2012 data so that it matches the 2013 data on some of the variables that may have been sensitive to the removal of the cap on student entry (or weight the 2013 data so that it matches the 2012 data). Then we can compare two different cohorts as if they were a proper control group for each other. In fact, this is illusory. Some suggested using SPSSINC RAKE, but I am not getting any clue why.

What we want to find out (very broadly) is whether introducing the advisers etc. worked, and the degree to which it worked.

Answer 1

Propensity score matching: First you'd model the probability of students' going to an advisor, based on available predictors—perhaps course marks, financial situation, living arrangements, medical records, &c. For each student the predicted probability of going to an advisor is the propensity score. Next you'd match each student who did go to an advisor with one who didn't but has the same, or as near as possible, propensity score. So you'd now proceed with an analysis of retention based on matched treatment-control pairs.

Of course, this approach doesn't provide as compelling evidence as an experiment in which students are randomly assigned to go to an advisor or not—you'd need to consider whether predictors you hadn't accounted for could be significantly contributing to the effect.

Rosenbaum & Rubin (1983), "The Central Role of the Propensity Score in Observational Studies for Causal Effects", Biometrika, 70, 1

Rosenbaum & Rubin (1985), "Constructing a Control Group Using Multivariate Matched Sampling Methods that Incorporate the Propensity Score", The American Statistician, 39, 1

And the Matching package for R might be useful.

Answer 2

A question about your data: Do students visit the advisers more than once? What I mean is, for the students who choose to go to the advisers, do some students visit their adviser once while others choose to visit 15-20 times? If this is the case, then I would advise against using the traditional propensity score method as laid out by Rosenbaum and Rubin, and instead look at the Generalized Propensity Score as discussed by Hirano and Imbens (2004) "The Propensity Score with Continuous Treatments".

Also, I feel the need to insert an obligatory "correlation is not causation" statement here because of your observation that "Just looking at 2013 data alone makes it seem that the advisers make things worse: the students they see do worse, and are more likely to leave university." Something I see a LOT in education data is that, whenever students are free to self-select into a treatment, the types of students who select into that treatment are very different than those who don't, and often there is huge variation even among the students that do select into treatment. This makes evaluating the impact of an optional treatment program very difficult!