Testing for feature importance with missing values

Question

I'm looking for an appropriate model to do the following analysis:

I'd like to test which courses are the most important in determining if a student stays in or leaves a university program. Imagine I have a dataset that looks like:

target (y) = a 0,1 variable indicating if a student left the program.

Features: There are 10 features and each feature is a student's GPA relative to the mean in one of 10 courses. (Imaging we can somehow limit the possible "important" courses to these 10.) Courses are taken in sequence, though that sequence varies across students.

I was imaging using something like Sklearn's Recursive Feature Elimination or ExtraTreesClassifier to identify the most important features. The problem is that for the students who left to program many of the feature values will be missing. (For instance, if a student drops out before taking MATH200 then that value will be missing.)

If I use some sort of value to indicate a student didn't take a course, then I'll have data leakage (since not taking the course indicates that the student dropped out).

Could someone share their opinion as to how I could test for the most important features given the source of missing values?

Answer 1

This is not (yet) a solution, but I find the problem interesting and would like to share my thoughts on it.

Intuitively, you could look at students at certain points in their studies (e.g. end of 2nd semester, end of 3rd semester, ...) - using only information already available then to predict if they made it through the program in the end. This would give you the same NA information for both students that made/did not make it and remove the information leak.

Why would I try this? Because, as you said, there is a clear semantic connection between having many missing information and leaving early. This means not having much missing information indicates the student probably made it through the program - which I would consider to be semantically correct (and if such a relation is there and available, each model should use it...). Note that this information is a combination of information about multiple classes, but does not consider the performance in classes yet - it only uses the fact that this information exists at all. So, as you said, instead of selecting one/more well performing feature, this is just "counting features".

Though students can chose the order of courses, I guess that they will still have something like a fuzzy order in the courses they take (like some being more in the beginning of studies and some more towards the end). If this is true, passing a course that's usually taken towards the end will transport different information about the student making it through the program (but at the same time I fear that this might not be too helpful for your goal, as you might want to predict earlier if a student makes it through the program, right?) Anyway, this would probably be another argument for looking at students at a certain point in time, using all information available then to to predict if they made it through the program in the end.

Concerning feature importance: if you look only at data about students available at such a certain point in time, I'd guess that you could use standard mechanisms for feature importance, feature-target-correlation and feature selection again.