In 2012 we collected data at our university about retention of students from first semester to second semester along with some other variables. The retention variable is binary, 'retained' or 'did not retain'.
In 2013 we introduced a new system. Student success advisers were assigned and any student when needed could go to them. We collected some new variables associated with the introduction of the student success advisers together with those collected in 2012. Now, going to these advisers was not mandatory. Those who felt a problem could go to them.
Just looking at 2013 data alone makes it seem that the advisers make things worse: the students they see do worse, and are more likely to leave university. Only the students who faced problems probably went to the advisers and those who were doing well probably didn't. If 10 students went to the advisers with problems and 8 retained to semester 2, that is surely a success. But if 80 out of 100 students who didn't have any problem and thus didn't go to the adviser retained to semester 2, that will have log odds similar to those who went to the advisers and thus tell you that going to the advisers had no significant effect!
Part of the reason was a major policy change that affected how universities in our country took in students. There was a "cap" in place before 2013, a limit on how many students a university could take on and expect government funding for. In 2013 that was removed. So, successful, desirable universities started opening their doors to more students, and more students were able, suddenly, to get into desirable universities. Our university is not so desirable. So what probably happened was that our university was forced in 2013 to take significantly "worse" students, which could well lead to greater retention difficulty.
What my bosses think is that we need to weight the 2012 data so that it matches the 2013 data on some of the variables that may have been sensitive to the removal of the cap on student entry (or weight the 2013 data so that it matches the 2012 data). Then we can compare two different cohorts as if they were a proper control group for each other. In fact, this is illusory. Some suggested using SPSSINC RAKE, but I am not getting any clue why.
What we want to find out (very broadly) is whether introducing the advisers etc. worked, and the degree to which it worked.