How to properly perform predictions in ordinal regression?

Question

This question is a sequel to this one.

Proportional odds logistic regression predicts probabilities for each level $l$, conditioned on the predictor $x$: $$ P(y = l ~|~ x) \text{ for every } l \in L $$

But in practice we mostly simply want to predict the level $l$ itself. I recon the standard way is to pick the most probable level for $x$. This at least seems to be the default way how predict for R's MASS::polr works.

An alternative is to compute the expectation of the level and round it: $$ y = \lfloor ~ \sum_{l \in L} l \cdot P(y = l ~|~ x) ~ \rceil $$ (as written, this works only for levels enumerated as numbers, e.g. $0, 1, ...$, but it is trivial to extend it to other values).

The two approaches differ and produce different predictions. I'd assume the first one, as implemented in standard statistical software, should be preferred, but, at least on my dataset from the previous question, the second one performs better (MAE = 0.97 vs. MAE = 1.71) and approaches the performance of the binomial predictor (MAE = 0.94). Notice how picking the most probable level never selects levels 1 and 2:

So, which is the "standard" way of predicting the level and under which circumstances can an alternative be justified?

Edit in response to comment: I'm implicitly assuming that the ordinal scale is a discretised version of an underlying, latent, continuous variable, as suggested in McCullagh's (1980) original paper.

Answer 1

The reason why picking the most probable category is standard is that it doesn't involve any treatment of ordinal categories as if they were interval scaled. Computing the expectation treats the ordinal categories as numbers. The MAE does the same, so it is no surprise that the expectation gives better results using the MAE as a loss. The most probable category will normally give better results if as a loss you just count how often you get the category wrong (this is not a theorem and may occasionally not hold, at least not on test or validation data). So the way you pick the predicted category is related to what loss is relevant for you.

Arguably neither MAE nor misclassification probability are 100% appropriate, one implicitly treats the data as having interval level, the other one as having nominal level. There are loss functions specifically for ordinal data in the literature, but chances are they are rarely used, and there's surely more than one way of defining them, potentially leading to ambiguous results.

Ultimately it's your call what kind of loss is relevant for you. There's a dogma that ordinal data should not be used to do interval scaled stuff such as computing means, expectations, or MAE, however in a number of applications treating the categories as interval scaled numbers seems appropriate, particularly if there is an underlying quantitative scale that is equally split into ordinal categories (just assuming that there's an underlying scale without knowing what it is doesn't help though, because the categories might represent differently large intervals of values, in which case using the category numbers as interval scaled would misrepresent the underlying scale), or where data come from questionnaires that implicitly communicate to the respondents that data will be evaluated as integer numbers (for example if such numbers are explicitly given). If you actually decide that the MAE is most relevant for you, nothing should stop you from using the expectation or whatever is best to optimise your validation loss, however it may then well be that an ordinal regression is not the best method to start with in the first place. If you ultimately ignore the ordinal character of you data, why bother running a regression that is based on it (although if the model is close to the truth it may work well)?

Whether the misclassification loss is appropriate depends on whether in your specific application it is relevant if a prediction is wrong how wrong it actually is. Once more, if this is what you decide, methods for (categorical) classification can be competitive against ordinal regression.

A key issue is that any definition of a loss function will somehow quantify what according to the basic idea of ordinal data should not be quantified. You can somehow estimate the quantification from the data (looking at empirical frequencies of the categories, although it is problematic to assume that these are informative about "true" quantitative differences between categories), but ultimately, if you want to quantify loss, there is no way around having one. And if the loss is what you are most concerned about, pick the method that optimises yours.

Answer 2

When Y is discrete we are really stuck:

• The most probable category may not be very probable, so using it may result in an arbitrary forced choice
• If Y is not numeric and is not approximately interval-scaled then you can't use the mean
• Quantiles work only for continuous distributions (they are too jumpy when the distribution is discrete; you may find that adding a single observation moves a quantile a whole Y level)

So we are left with

• Estimating the whole probability distribution of Y given X, which is fine but is just noisy
• Estimating P(Y=y | X) which is fine, but will have low precision
• Estimating P(Y $\geq$ y | X) for pre-specified y, which is fine