# How to properly perform predictions in ordinal regression?

This question is a sequel to this one.

Proportional odds logistic regression predicts probabilities for each level $$l$$, conditioned on the predictor $$x$$: $$P(y = l ~|~ x) \text{ for every } l \in L$$

But in practice we mostly simply want to predict the level $$l$$ itself. I recon the standard way is to pick the most probable level for $$x$$. This at least seems to be the default way how predict for R's MASS::polr works.

An alternative is to compute the expectation of the level and round it: $$y = \lfloor ~ \sum_{l \in L} l \cdot P(y = l ~|~ x) ~ \rceil$$ (as written, this works only for levels enumerated as numbers, e.g. $$0, 1, ...$$, but it is trivial to extend it to other values).

The two approaches differ and produce different predictions. I'd assume the first one, as implemented in standard statistical software, should be preferred, but, at least on my dataset from the previous question, the second one performs better (MAE = 0.97 vs. MAE = 1.71) and approaches the performance of the binomial predictor (MAE = 0.94). Notice how picking the most probable level never selects levels 1 and 2:

So, which is the "standard" way of predicting the level and under which circumstances can an alternative be justified?

Edit in response to comment: I'm implicitly assuming that the ordinal scale is a discretised version of an underlying, latent, continuous variable, as suggested in McCullagh's (1980) original paper.

• You treat levels as numbers (rather than categories) in obtaining the second prediction and subsequently in model evaluation (obtaining MAE). You treat them as categories in the modelling phase (use logistic regression, ordered logit or something like that). Does that not seem problematic to you? Oct 7, 2021 at 16:00
• @RichardHardy At first glance, you seem to have a point. But, if we assume "latent variable model" behind ordinal regression, don't you think that by treating categories as numbers in the evaluation phase I tend to rectify the error (or attempt to recover the information lost) from the modelling phase? Oct 8, 2021 at 6:24
• I am not familiar with the latent variable model behind ordinal regression from before. On a quick glance, it seems it assumes more structure than raw ordinal regression does. That additional structure may be driving your argument. If you assume it, I suggest you make it explicit in your post. (But I may be wrong.) Oct 8, 2021 at 6:54
• Thanks, @RichardHardy, I updated the post. Just out of curiosity: Can you point me to a practical example of ordinal regression where assuming a latent continuous variable would make no sense? Oct 8, 2021 at 9:10
• I don't see the helpfulness of thinking about latent variables, unless you want to extend the model in strange ways. There is a direct way to state ordinal regression models. Oct 8, 2021 at 12:12

The reason why picking the most probable category is standard is that it doesn't involve any treatment of ordinal categories as if they were interval scaled. Computing the expectation treats the ordinal categories as numbers. The MAE does the same, so it is no surprise that the expectation gives better results using the MAE as a loss. The most probable category will normally give better results if as a loss you just count how often you get the category wrong (this is not a theorem and may occasionally not hold, at least not on test or validation data). So the way you pick the predicted category is related to what loss is relevant for you.

Arguably neither MAE nor misclassification probability are 100% appropriate, one implicitly treats the data as having interval level, the other one as having nominal level. There are loss functions specifically for ordinal data in the literature, but chances are they are rarely used, and there's surely more than one way of defining them, potentially leading to ambiguous results.

Ultimately it's your call what kind of loss is relevant for you. There's a dogma that ordinal data should not be used to do interval scaled stuff such as computing means, expectations, or MAE, however in a number of applications treating the categories as interval scaled numbers seems appropriate, particularly if there is an underlying quantitative scale that is equally split into ordinal categories (just assuming that there's an underlying scale without knowing what it is doesn't help though, because the categories might represent differently large intervals of values, in which case using the category numbers as interval scaled would misrepresent the underlying scale), or where data come from questionnaires that implicitly communicate to the respondents that data will be evaluated as integer numbers (for example if such numbers are explicitly given). If you actually decide that the MAE is most relevant for you, nothing should stop you from using the expectation or whatever is best to optimise your validation loss, however it may then well be that an ordinal regression is not the best method to start with in the first place. If you ultimately ignore the ordinal character of you data, why bother running a regression that is based on it (although if the model is close to the truth it may work well)?

Whether the misclassification loss is appropriate depends on whether in your specific application it is relevant if a prediction is wrong how wrong it actually is. Once more, if this is what you decide, methods for (categorical) classification can be competitive against ordinal regression.

A key issue is that any definition of a loss function will somehow quantify what according to the basic idea of ordinal data should not be quantified. You can somehow estimate the quantification from the data (looking at empirical frequencies of the categories, although it is problematic to assume that these are informative about "true" quantitative differences between categories), but ultimately, if you want to quantify loss, there is no way around having one. And if the loss is what you are most concerned about, pick the method that optimises yours.

• Thanks for pointing out the connection between MAE and expectation. Your scepticism is justified, but isn't treating levels as nominal variables even worse? E.g. isn't giving a 'D' to a student who deserves an 'A' worse than giving him/her a 'B'? Oct 12, 2021 at 6:40
• @IgorF. It all depends on context. What you say in the comment is true in the given situation. However, you're talking about "giving" a student a mark far off here, not "predicting" it, which is what the question is about. Obviously one can argue that if it's a problem with giving, it's also a problem with predicting, but they're not exactly the same and it depends on all specifics. Student grades are sometimes averaged and summed up in numerical ways, determining student's prospects, so arguably then nominal treatment is worse than numerical, but that's not a general statement. Oct 12, 2021 at 10:09
• @IgorF. Also note that in my answer I'm not advertising any particular loss function. I say that the researcher, knowing the situation in detail, needs to decide this. If you in your application decide and can argue well that MAE is appropriate, that's all fine and in line with my answer. Oct 12, 2021 at 10:13

When Y is discrete we are really stuck:

• The most probable category may not be very probable, so using it may result in an arbitrary forced choice
• If Y is not numeric and is not approximately interval-scaled then you can't use the mean
• Quantiles work only for continuous distributions (they are too jumpy when the distribution is discrete; you may find that adding a single observation moves a quantile a whole Y level)

So we are left with

• Estimating the whole probability distribution of Y given X, which is fine but is just noisy
• Estimating P(Y=y | X) which is fine, but will have low precision
• Estimating P(Y $$\geq$$ y | X) for pre-specified y, which is fine
• So, if I understand you correctly, my desired output - the level which corresponds to given X - is not really doable in ordinal regression? Oct 12, 2021 at 6:35
• There is no such thing as "the level which corresponds to a given X" unless the model provides perfect predictions. In the discrete ordinal case I'd stick with estimated cumulative probabilities (exceedance probabilities) $P(Y \geq y | X)$ for a variety of values of $y$ and $X$. Oct 12, 2021 at 11:54
• Another way of putting things is that your desired output is not very sensible unless the highest probability value of Y has probability > 0.8 or so. Oct 15, 2021 at 13:07