Edit: Shared my solution as an answer here
Our goal is to determine optimal cut-off test scores for course placement. The course placement has already been manually assigned to each test-taker. The goal is to replace this manual labor with the calculated cut-off test scores, so that a future test-taker from a similar group will be automatically placed into an optimal course.
We're looking for cut-off scores such as this:
- 0-9: Course A
- 10-14: Course B
- 15-19: Course C
- 20-24: Course D
- 25-30: Course E
In this example, if a student answers 14 questions correctly, they'd be placed into Course B.
The variables in this analysis are
- Independent: The screening test score, which is a continuous variable ranging from 0-30
- Dependent: Course placement, which is an ordinal variable, or a categorical variable for which there is a clear ordering of the category labels (i.e. Course B is more advanced than Course A, Course C is more advanced than Course B, and so forth).
Approaches we've considered, along with our concerns about them:
- Set the cut-off score for a specific course at one standard deviation above the median score for all the students who got placed into that course. (This uses the median instead of the mean, because there are some outliers in the data where some students who scored really high got placed into a low-level course).
- Concerns: Imbalanced data. One of the courses had a disproportionately higher number of students placed into it, which inflates the accuracy of placement.
- Ordinal logistic regression — We've used this model to obtain the probabilities of being in a specific course, given a test score.
- Concerns: These are probabilities and some of them overlap equally, so how can we decide with certainty which score value the cutoff should fall on? Is regression the correct approach?
How would you recommend we go about creating cut-off scores for this course placement test and evaluating it?