Some categories never predicted in ordinal logistic regression model

Question

SUBJECT: Some of the predicted categories missing in the ordinal logistic regression output

In my ordinal logistic regression model, I have a set of 7 inputs and I have Y = 2, 3, 4, ..., 19 (18 categories) as my response variable. I use SAS PROC LOGISTIC and I am using the output probabilities to classify each observation in to a specific category depending on which probability of the 18 probabilities output by SAS is the maximum. For example, if p10 is the highest probability of all 18 probabilities, then, I classify this observation in to category 10. Just to double check, I also use the estimates output by proc logistic and do explicit logistic calculations as follows:

cum prob, c1 = exp(&inter1 - modelScore)/(1 + exp(&inter1 - modelScore))
...
cum prob, c17 = exp(&inter17 - modelScore)/(1 + exp(&inter17 - modelScore)); 

and calculate the individual probabilities p1 to p18 with p1 = c1, p2 = c2 - p1, and p18 = 1 - c17.

The problem is, the categories (2, 3, 6, 7, 9, 10, 16, 18) are never occurring in the predicted response. Can someone explain to me what I might be doing wrong? Whether I use SAS probabilities or the explicitly calculated probabilities, the same rating categories are missing in the predicted response although the Y response variable has the entire spectrum from 2 to 18.

Answer 1

You predict a class if it is given maximum probability by the estimated model. So when some classes are not predicted that is simply because the model never gave them maximum probability. That might just be correct, and not necessary a reason for concern. So I disagree with the answer by @Nitin, proposing oversampling.

You might say: but they did occur in the data. Yes, but it might have been rare occurrences! never really the most probable outcome given the predictors in the model. You didn't give us a context, what your classes represent in the "real world". You have very unbalanced classes. That might be because some classes really are uncommon in your population, or it might be some problems with data collection. You didn't tell us. But it is difficult to see that over (or under)-sampling can achieve anything that cannot be achieved using weights.

Even more important, you are using (ordinal) logistic regression, which is not a classifier, see Why isn't Logistic Regression called Logistic Classification?. Logistic regression gives you estimated probabilities for class membership, and instead of just looking at the maximum predicted probability, you could compare the predicted probabilities with the population proportions. Or even pass to use some proper scoring rule. See Using proper scoring rule to determine class membership from logistic regression or Is accuracy an improper scoring rule in a binary classification setting?.

About the use of over/under-sampling, our community member @Frank Harrell have on this site (and elsewhere) commented against its use, see Downsampling vs upsampling on the significance of the predictors in logistic regression and https://www.fharrell.com/post/class-damage/.

Answer 2

Here's a nice paper i found, while facing the same problem: http://www.ele.uri.edu/faculty/he/PDFfiles/ImbalancedLearning.pdf

Edited to summarize: The paper gives the following approaches to deal with imbalanced datasets, largely focusing on binary classification examples:

a) Simple Oversampling minority classes & Undersampling majority classes

b) Informed Undersampling - e.g undersample majority class, combine with minority data to create dataset, apply a classification model, repeat the exercise for N-samples and models, and bag the results

c) SMOTE and Adaptive SMOTE to artifically create new samples of minority class and then apply Tomek Links to remove pair instances of majority+minority classes near the borders

d) Cost based methods with boosting to incorporate cost parameters that bias the minority class for boosting learner models

e) An overview of validation methods, e.g Accuracy vs ROC Curve vs PR curve vs Cost Curves

I am still experimenting with these methods for my own problem: the informed undersampling with bagging seems a nice approach for a beginner such as myself.

My problem is quite similar to that of the original question: I have 8 unbalanced classes in my dataset, and both my GAM model specifying ordered categorical family as initial distribution (gam & ocat from R's mgcv package, more info here: https://cran.r-project.org/web/packages/mgcv/mgcv.pdf) and my ordinal regression model both are biased towards the same 4 set of classes and completely ignore the under-represented classes.