0
$\begingroup$

Edit: Shared my solution as an answer here

Our goal is to determine optimal cut-off test scores for course placement. The course placement has already been manually assigned to each test-taker. The goal is to replace this manual labor with the calculated cut-off test scores, so that a future test-taker from a similar group will be automatically placed into an optimal course.

We're looking for cut-off scores such as this:

  • 0-9: Course A
  • 10-14: Course B
  • 15-19: Course C
  • 20-24: Course D
  • 25-30: Course E

In this example, if a student answers 14 questions correctly, they'd be placed into Course B.

The variables in this analysis are

  • Independent: The screening test score, which is a continuous variable ranging from 0-30
  • Dependent: Course placement, which is an ordinal variable, or a categorical variable for which there is a clear ordering of the category labels (i.e. Course B is more advanced than Course A, Course C is more advanced than Course B, and so forth).

Approaches we've considered, along with our concerns about them:

  • Set the cut-off score for a specific course at one standard deviation above the median score for all the students who got placed into that course. (This uses the median instead of the mean, because there are some outliers in the data where some students who scored really high got placed into a low-level course).
    • Concerns: Imbalanced data. One of the courses had a disproportionately higher number of students placed into it, which inflates the accuracy of placement.
  • Ordinal logistic regression — We've used this model to obtain the probabilities of being in a specific course, given a test score.
    • Concerns: These are probabilities and some of them overlap equally, so how can we decide with certainty which score value the cutoff should fall on? Is regression the correct approach?

enter image description here

How would you recommend we go about creating cut-off scores for this course placement test and evaluating it?

Improve this question
$\endgroup$
14
  • 2
    $\begingroup$ To clarify, your data set currently is the score on the screening test and the class they were placed into manually (i.e., regardless of their score on the screening test). ¿Do you have any additional data? For example, ¿do you have the scores/passing rates for how the students performed in their assigned classes? $\endgroup$
    – Gregg H
    Commented Jun 3, 2023 at 20:53
  • 4
    $\begingroup$ Your problem is impossible to solve because its formulation lacks the information needed to understand and operationalize the sense of "optimal." One would think optimality might be framed in terms of the greatest net good for the student population, but how exactly is that determined? How do you propose to trade off the harm in placing a student in an inappropriate course for the benefit of placing other students in appropriate courses? Do you account for any characteristics of the course composition, such as diversity of thought or background? What constraints on course size are there? $\endgroup$
    – whuber ♦
    Commented Jun 7, 2023 at 22:32
  • 3
    $\begingroup$ "which score value the cutoff should fall on? " On what basis should we be able to decide on these cutoff values? Do you regard the manual placement as a golden standard? Why do your curves show students with the best test scores being placed in the second lowest level course? You could choose some cutoff values, but in any case it seems like you will get a lot different composition of the courses in comparison to your manual process. $\endgroup$
    – Sextus Empiricus
    Commented Jun 7, 2023 at 22:53
  • 1
    $\begingroup$ That's a start. But to make any progress you need to connect -- quantitatively -- the test score to some measure of the learning experience. Absent any connection, an optimal algorithm is to assign students arbitrarily to the classes. Moreover, if the class sizes are flexible, you also need a quantitative measure of the cost associated with possible class sizes (and that must be commensurate with the learning experience measure). $\endgroup$
    – whuber ♦
    Commented Jun 9, 2023 at 12:10
  • 1
    $\begingroup$ @anneirb The use of ROC curves is more a situation when you have positive/negative cases and you want to optimize the benefit of true positives versus the costs of false positives. In your example you do not specify anything that could resemble the relative costs of misclassification in the different classes.... $\endgroup$
    – Sextus Empiricus
    Commented Jun 9, 2023 at 14:54

6 Answers 6

Reset to default
3
$\begingroup$

, it may be possible to find an optimal cut-off if we used a ROC curve here somehow, to prioritize specificity over sensitivity

Sensitivity and specificity is a consideration when there is a binary classification and we consider the ability to recognize a particular single class.

Your situation is not exactly like that, as you are considering the classification of multiple classes. Increasing the sensitivity of one class will decrease the sensitivity of the other classes.

For example, when you shift the lowest boundary and increase the number of placements into the PST301 class, then this increases the sensitivity detecting the students of the PST301 class, but it is decreasing the sensitivity of all the other classes.

In order to make an optimization you need some quantitative expression to optimize. This expression should help to answer questions like: What are the pro's and con's of misclassifying a particular class and how do they compare? Is it worth to increase the sensitivity of detecting students for class X at the cost of decreasing the sensitivity of students in class Y?

These questions need to be answered and expressed in a quantitative way, before you can optimize the boundaries.

Otherwise, if all things are equal then just use the following classification:

0-10: Course A
11-17: Course B
18-30: Course D

removed courses
Course C
Course E

This will give the highest number of correct classifications. Any shift of the boundaries will result in less correct classifications.

Choosing to go for less correct classifications might be an option, but it depends on the considerations that quantifies the value of the errors in different classifications. (e.g. if the course A is cheaper then you might place more people in that course, even when it is misclassifying more students in total)

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ And there can be more complex considerations. (1) the cost function can be more complex, like considerations as the optimal class sizes which Whuber mentioned in the comments (you say that this is not the case, but it is just an example) (2) the statistical analysis can be more complex if you consider measurement errors, changes in time, etc. $\endgroup$
    – Sextus Empiricus
    Commented Jun 9, 2023 at 15:32
  • $\begingroup$ Your comments about a cost function are helpful. By prioritizing specificity, we aim to minimize the false positive rate, or overplacing students into much higher-level courses. Misclassifying course A into course C should be worse than misclassifying course A into course B. It seems that a cost function that captures this definition of "optimal" may help determine clearer boundaries. Would this cost function be applied during ordinal logistic regression, or to the results? $\endgroup$
    – anneirb
    Commented Jun 9, 2023 at 20:52
  • $\begingroup$ @anneirb the problem is also the classification of classes next to each other. How do you compare and quantify misclassifying 'course A into course B' versus misclassifying 'course B into course A'? $\endgroup$
    – Sextus Empiricus
    Commented Jun 9, 2023 at 20:57
  • $\begingroup$ @anneirb the terms specificity and sensitivity are not very clear in this problem. The increase of sensitivity for one category is the decrease of sensitivity for another category. There is no clear way how you wish to combine the sensitivity of different courses. $\endgroup$
    – Sextus Empiricus
    Commented Jun 9, 2023 at 21:00
  • 1
    $\begingroup$ "That's pretty much the question in this post" A problem with it is that it is not stated in detail. The question currently reads like "I have a problem that involves a parameter $x$. How do I optimize $x$?" without telling what it is that needs to be optimized. You might say that the cut-off values are the object that need to be optimized, but those values are just the parameters that can be adjusted to perform the optimization. The object that is to be optimized should be some quantitative function of the cut-off values. $\endgroup$
    – Sextus Empiricus
    Commented Jun 9, 2023 at 21:57
2
+50
$\begingroup$

If courses A, B, C, D, E got $a,b,c,d,e$ manual assignments among the $N=a+b+c+d+e$ students so far, then as a first attempt I would:

  • set the upper cutoff for course A at the $a^{th}$-worst of the scores so far;
  • set the upper cutoff for course B at the $(a+b)^{th}$-worst of the scores so far;
  • set the lower cutoff for course D at the $(d+e)^{th}$-best of the scores so far;
  • set the lower cutoff for course E at the $e^{th}$-best of the scores so far.

So far I don’t see a reason for anything more complicated than that.

Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ It seems plausible to use the quantiles of the test scores such that a particular ratio of class sizes is obtained. But the last comment 'no reason for anything more complicated' is not so clear. Why is it that these percentiles are optimal? What optimization is performed with it? For all such questions and for the actual underlying cost function to be optimized (which is not stated explicitly in the question but we can see all sorts of reasons for various kinds of approaches with different cost functions), for all that we have reasons for something more complicated. $\endgroup$
    – Sextus Empiricus
    Commented Jun 11, 2023 at 10:33
  • $\begingroup$ I guess this optimized for simplicity. The question omitted the complications that seem likely to me: considering the student’s intended degree, time at the school so far, scores on different parts of the test, etc. $\endgroup$
    – user225256
    Commented Jun 11, 2023 at 12:16
  • 1
    $\begingroup$ We don't want to base the cutoffs on class sizes or to have them controlled by that in any way. But I'd also like to optimize for simplicity. This looks like an approach to evaluate, as it gives a cut off of A: 0-11, B: 12-15, C: 16-20, D: 21-27, E: 28> $\endgroup$
    – anneirb
    Commented Jun 13, 2023 at 14:42
  • $\begingroup$ How would this translate to other number of courses, e.g. if we had 3 courses instead of 5? $\endgroup$
    – anneirb
    Commented Jun 14, 2023 at 2:02
  • 1
    $\begingroup$ For three courses A, B, C with $a, b, c$ manual placements, I would set the upper cutoff for course A at the $a^{th}$-worst score and the lower cutoff for course C at the $c^{th}$-best score. Meanwhile the metric of accuracy is up to you! $\endgroup$
    – user225256
    Commented Jun 14, 2023 at 8:36
1
$\begingroup$

My suggestion is the following:

  1. using a logistic regression of the category regressed on the score of the test. Find the 50% cut-off for A vs B-or-C-or-D-or-E
  2. repeat but find the 50% cut-off for A-or-B vs C-or-D-or-E
  3. repeat this process for all cumulative boundary points

But, do not stop here. Next, obtain information on how well these boundary assignments work for the next batch of students. For example, ¿did they fare well in their assigned course? You can revisit the cutoffs using this information.

Finally, if you are so inclined, you can also do an analysis that looks at each question, and how well that question correlates (is associated) with the total score (and by extension the cut-off scores). If there are certain questions that demonstrate high discrimination, these items can be highlighted for later use as "drop-dead" boundaries. These are separate from the thresholds, and are must-pass questions to make to a certain level.

Improve this answer
$\endgroup$
0
$\begingroup$

1- If you predict you ordinal probit/logit modelinto you population, you will get scores of each group, and then you can make a rule based on the highest proba per group, in case of matching proba scores (i dont thik this will happen) you send up or down (randomly? or based on an specific question)
2-It seems that you are also concerned about the number of students in each group. So if this is an issue, why don't you create deciles/xtiles and assign each person to their groups? This solution, will create equal number of members. This approach could be complemented with other rules, such as score below 25 do not get to go to class X and so forth.

hope it helps,

Improve this answer
$\endgroup$
1
  • $\begingroup$ We're not concerned about the number of students in each group. We don't want to force a student into an inappropriate course level just to fill seats. $\endgroup$
    – anneirb
    Commented Jun 9, 2023 at 14:05
0
$\begingroup$

Don't use the aggregate score. There would usually be some gradation of 'hardness' in the questions that will add extra information to your analysis and there might also be some utility in matching the subject matter of the questions with the course options.

As a retired university academic I can say from experience that if you look at the relationships between success on each question and the aggregate score, and with indices of prior academic success and you will almost certainly find some interesting patterns that could help you in your allocations goal. You might also find occasionally a question that predicts backwards (i.e. a question where success correlates with lower performance overall), and bad questions that have no truly correct answer or more than one correct answer.

Before you do that, however, make sure that you define the criteria for 'good' and 'bad' allocations. Without clearly thought out criteria you cannot even define 'optimal', much less achieve it.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Instead of using the aggregate score, would you recommend using each individual question in the logistic regression model, or are you referring to using them in a different way? Just trying to understand exactly how you'd not use the aggregate score. Also noting that we've done Cronbach's alpha and other internal consistency tests to weed out some bad questions, and we observe some weak patterns in the relationship between the question and the aggregate score. $\endgroup$
    – anneirb
    Commented Jun 14, 2023 at 22:30
  • $\begingroup$ I am suggesting that you should be using the available information rather than just using the easy numbers. Success in a hard question should count more towards the final index of academic success than success in a question that almost everyone gets right. Also note that some wrong answers are 'more wrong' than others; some represent failure to understand and some represent a failure to notice a clue or a trick. You need subject matter expert input for the meanings of each right and wrong answer for each question. $\endgroup$
    – Michael Lew
    Commented Jun 15, 2023 at 2:02
  • $\begingroup$ I agree that it would be ideal to use the available information, especially since some questions may be "harder" than others, and thus more correlated with higher courses, or vice versa. I'm curious about how you'd implement what you're suggesting. $\endgroup$
    – anneirb
    Commented Jun 15, 2023 at 2:32
  • $\begingroup$ How to implement it depends on what information you have and what you can distill from the data. At one end you might simply use a weighted aggregate score for your thresholding, where the weights reflect question hardness. At the other end you would use information about the conceptual strengths and skills to allocate students to courses that suit them or would help them to overcome weaknesses. There is no statistical method for what you want to do. $\endgroup$
    – Michael Lew
    Commented Jun 15, 2023 at 2:55
0
$\begingroup$

I ended up making a cost function, generated a comprehensive list of potential cut-off scores that were at least $x$ points apart (to avoid overly sensitive course placements), and evaluated them all using the cost function.

The criteria is

  • Avoid assigning a student to an overly advanced course.
  • Avoid assigning a student to a course that doesn’t challenge them enough.
  • If a student must be assigned to a course that doesn’t fit their abilities, it’s better to assign them to the next immediate level rather than skipping multiple levels. It’s also better to lean towards a lower level than a higher one.
  • Adjust for class sizes, as some classes have had a significantly larger student count. Without this weighting, the larger courses will dominate the optimization process.

The cost function is

enter image description here

where for each set of cut-off scores that I evaluate, I assign each student $i$ a penalty $c_i$ that is weighted by the inverse of the size of their actual course placement $w(p_i)$. I sum the penalties across all $N$ students to get the cost $J$ of the cut-off scores. Lower costs indicate more optimal cut-off scores.

The penalty $c$ is calculated using

enter image description here

where I give each student a penalty that is based on the level difference Δ between their actual course placement ($p_i$) and the course placement predicted by their score ($d_i$). No penalty is given if both placements match. When the cut-off score suggests a higher course level than the student’s actual placement, the penalty is the square of the level difference, increasing as the course level increases. Conversely, if the score indicates a lower course level, the penalty is half the absolute difference, also increasing as the course level decreases. The penalty is steeper for upward misplacements, emphasizing the undesirability of placing a student into a course that is too high for their abilities.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Why do you add a weight $w(p_i)$ that is related to the class sizes? Is there a problem with small class sizes and if there's a problem how does that problem work in creating costs such as you describe? It seems a bit weird to apply such rule as, at the moment the current class sizes don't seem equal either. But also, it is not well expressed how class size should be taken into consideration. Is it a problem with the salary of the teachers, planning the roster, the sizes of the rooms? When I look at the graphs I would think that it is beneficial to just eliminate some classes. $\endgroup$
    – Sextus Empiricus
    Commented Aug 25, 2023 at 10:36
  • $\begingroup$ I have made that recommendation, as the class sizes are not equal and there were only 2 students placed into one course, and the university cancels classes that have <6 students. For this round of cutoff score analysis, the weights help ensure that every class has a chance of being represented in the optimization process, regardless of whether they should be dropped, as that decision is not my call to make. $\endgroup$
    – anneirb
    Commented Aug 25, 2023 at 14:23
  • $\begingroup$ Do let me know if you have any further thoughts. $\endgroup$
    – anneirb
    Commented Aug 25, 2023 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.