# Comparing the effect of a treatment that was optional for its receivers

In 2012 we collected data at our university about retention of students from first semester to second semester along with some other variables. The retention variable is binary, 'retained' or 'did not retain'.

In 2013 we introduced a new system. Student success advisers were assigned and any student when needed could go to them. We collected some new variables associated with the introduction of the student success advisers together with those collected in 2012. Now, going to these advisers was not mandatory. Those who felt a problem could go to them.

Just looking at 2013 data alone makes it seem that the advisers make things worse: the students they see do worse, and are more likely to leave university. Only the students who faced problems probably went to the advisers and those who were doing well probably didn't. If 10 students went to the advisers with problems and 8 retained to semester 2, that is surely a success. But if 80 out of 100 students who didn't have any problem and thus didn't go to the adviser retained to semester 2, that will have log odds similar to those who went to the advisers and thus tell you that going to the advisers had no significant effect!

Part of the reason was a major policy change that affected how universities in our country took in students. There was a "cap" in place before 2013, a limit on how many students a university could take on and expect government funding for. In 2013 that was removed. So, successful, desirable universities started opening their doors to more students, and more students were able, suddenly, to get into desirable universities. Our university is not so desirable. So what probably happened was that our university was forced in 2013 to take significantly "worse" students, which could well lead to greater retention difficulty.

What my bosses think is that we need to weight the 2012 data so that it matches the 2013 data on some of the variables that may have been sensitive to the removal of the cap on student entry (or weight the 2013 data so that it matches the 2012 data). Then we can compare two different cohorts as if they were a proper control group for each other. In fact, this is illusory. Some suggested using SPSSINC RAKE, but I am not getting any clue why.

What we want to find out (very broadly) is whether introducing the advisers etc. worked, and the degree to which it worked.

• Have you considered propensity-score matching for the 2013 students? – Scortchi - Reinstate Monica Nov 1 '13 at 10:56
• @Scortchi I don't have much idea about that. Could you please help by saying a little more? – Blain Waan Nov 4 '13 at 13:09

## 2 Answers

Propensity score matching: First you'd model the probability of students' going to an advisor, based on available predictors—perhaps course marks, financial situation, living arrangements, medical records, &c. For each student the predicted probability of going to an advisor is the propensity score. Next you'd match each student who did go to an advisor with one who didn't but has the same, or as near as possible, propensity score. So you'd now proceed with an analysis of retention based on matched treatment-control pairs.

Of course, this approach doesn't provide as compelling evidence as an experiment in which students are randomly assigned to go to an advisor or not—you'd need to consider whether predictors you hadn't accounted for could be significantly contributing to the effect.

Rosenbaum & Rubin (1983), "The Central Role of the Propensity Score in Observational Studies for Causal Effects", Biometrika, 70, 1

Rosenbaum & Rubin (1985), "Constructing a Control Group Using Multivariate Matched Sampling Methods that Incorporate the Propensity Score", The American Statistician, 39, 1

And the Matching package for R might be useful.

A question about your data: Do students visit the advisers more than once? What I mean is, for the students who choose to go to the advisers, do some students visit their adviser once while others choose to visit 15-20 times? If this is the case, then I would advise against using the traditional propensity score method as laid out by Rosenbaum and Rubin, and instead look at the Generalized Propensity Score as discussed by Hirano and Imbens (2004) "The Propensity Score with Continuous Treatments".

Also, I feel the need to insert an obligatory "correlation is not causation" statement here because of your observation that "Just looking at 2013 data alone makes it seem that the advisers make things worse: the students they see do worse, and are more likely to leave university." Something I see a LOT in education data is that, whenever students are free to self-select into a treatment, the types of students who select into that treatment are very different than those who don't, and often there is huge variation even among the students that do select into treatment. This makes evaluating the impact of an optional treatment program very difficult!

• Very good point: I'm sure many do go more than once, spend more time with them; seek, & obtain different types of help. – Scortchi - Reinstate Monica Nov 5 '13 at 22:19
• Thank you and +1 for both of you. As per your suggestion we do not really need the 2012 data. 2013 data can alone provide us with propensity scores. But my bosses still think we need to weight the 2012 data so that it matches the 2013 data on some of the variables (8 variables in total) that may have been sensitive to the removal of the cap on student entry (or weight the 2013 data so that it matches the 2012 data). Then we can compare two different cohorts as if they were a proper control group for each other. – Blain Waan Nov 7 '13 at 13:51
• They now simply want an answer along the lines of the following: "We weighted the 2013 cohort so that it was the same on key demographics as the 2012 cohort. In 2012, 42 (say) more students failed to return for the semester 2 2012 (attritition) than was the case with the matching dataset in 2013, presumably due to the interventions introduced in 2013." Is it possible?? Please help me with your kind suggestion (possibly in a different answer thread or by updating your existing answer). Thank you once again for your help. – Blain Waan Nov 7 '13 at 13:53
• Blain, you still haven't mentioned whether or not students who visit the student success advisers do so multiple times per year, or if it's just a one-time thing. Also, I'm confused about this notion of "weighting" cohorts - it sounds to me like you want a sample of the 2013 cohort that is statistically similar to a sample of the 2012 cohort, which is something that would be achieved anyway through the (correct) usage of propensity scores. – Nathan Calverley Nov 8 '13 at 20:18
• Actually I don't have information about how many times they visited the advisers but I have the number of hours they spent with the advisers. In 2012 we did not have advisers but we collected data on demographic variables of the students. In 2013 we introduced advisers and also collected information about the demographic variables. Now based on the similar demographic variables in the 2013 data, we want to make a comparison with the 2012 data on retention of the students. We also want to see if introducing advisers in 2013 had any effect on retention. – Blain Waan Nov 8 '13 at 22:27