# How to set up neural network to output ordinal data?

I have a neural network set up to predict something where the output variable is ordinal. I will describe below using three possible outputs A < B < C.

It is pretty obvious how to use a neural network to output categorical data: the output is just a softmax of the last (usually fully connected) layer, one per category, and the predicted category is the one with the largest output value (this is the default in many popular models). I have been using the same setup for ordinal values. However, in this case the outputs often don't make sense, for example the network outputs for A and C are high but B is low: this is not plausible for ordinal values.

I have one idea for this, which is to calculate loss based on comparing the outputs with 1 0 0 for A, 1 1 0 for B, and 1 1 1 for C. The exact thresholds can be tuned later using another classifier (eg Bayesian) but this seems to capture the essential idea of an ordering of inputs, without prescribing any specific interval scale.

What is the standard way of solving this problem? Is there any research or references that describe the pros and cons of different approaches?

• I got lots of interesting hits on Google for "ordinal logistic regression" e.g. this paper Commented Mar 3, 2015 at 1:53
• @ssdecontrol: Interesting. I tried it; the results were better than picking the one output with the highest value but slightly worse than other methods (naive Bayesian, etc). This is useful, but it doesn't help train the network, only improves results slightly after the fact... or at least I don't see how to make it help train the network. Commented Mar 3, 2015 at 8:22
• which "it" did you try? My only point is that the search engine could be more helpful than you might expect Commented Mar 3, 2015 at 8:56
• I also don't know how you could ever have an output like "1 1 0". I think there's some confusion about terminology here. Are you describing cumulative ordinal outcomes? As in a cumulative logit model? Commented Mar 3, 2015 at 9:11
• Commented Aug 1, 2018 at 21:49

I think the approach to only encode the ordinal labels as

• class 1 is represented as [0 0 0 0 ...]

• class 2 is represented as [1 0 0 0 ...]

• class 3 is represented as [1 1 0 0 ...]

and use binary cross-entropy as the loss function is suboptimal. As mentioned in the comments, it might happen that the predicted vector is for example [1 0 1 0 ...]. This is undesirable for making predictions.

The paper Rank-consistent ordinal regression for neural networks describes how to restrict the neural network to make rank-consistent predictions. You have to make sure that the last layer shares its weights, but should have different biases. You can implement this in Tensorflow by adding the following as the last part of the network (credits for https://stackoverflow.com/questions/59656313/how-to-share-weights-and-not-biases-in-keras-dense-layers):

class BiasLayer(tf.keras.layers.Layer):
def __init__(self, units, *args, **kwargs):
super(BiasLayer, self).__init__(*args, **kwargs)
shape=[units],
initializer='zeros',
trainable=True)

def call(self, x):
return x + self.bias

# Add the following as the output of the Sequential model


Note that the number of ordinal classes here is 5, hence the $$K-1$$ biases.

I tested the difference in performance on actual data, and the predictive accuracy improved substantially. Hope this helps.

• Neat!! I am searching for the resource to implement the original regression in PyTorch. "I tested the difference in performance on actual data and the predictive accuracy improved substantially." Do you mean the accuracy on the testing set? Which metric are you using? which models did you compare it with? Commented Nov 8, 2020 at 3:44
• I highly recommend this answer, I have been able to successfully implement without much additional work because of this repo: github.com/ck37/coral-ordinal Commented Dec 16, 2020 at 17:13
• If I understand the paper, then the Keras-based solution in the answer by @thijsvpd also guarantees the proper ordering of the biases, right? If so, this should be the top answer! Commented Jul 29, 2021 at 18:56
• @thijsvdp Have you encountered ordinal function with two-side ranking? For example, ordinal classes are [0, 1, 2, 3, 4] and 𝑃𝑎𝑟𝑔_𝑚𝑎𝑥=2, then it should be 𝑃(0)<𝑃(1)<𝑃(2)>𝑃(3)>𝑃(4) Commented Sep 29, 2021 at 10:14
• I guess this is not possible, because if I understand you correctly you have the following p0<p1<p2 and p4<p3<p2. This lacks information about the ordering of p0, p1, p3 and p4. For example, it can happen that p0<p4 or p4<p0. That information is not provided. I guess you can build a model which models both rankings separately. But if you want a single class prediction that cannot be done here because of lacking information between the ordering of p0 and p1 versus p3 and p4. Commented Sep 29, 2021 at 11:41

Update: Meanwhile, I delved more into the topic and even wrote some of the methods from the literature (see file losses.py). It includes the loss I mention here (ordinal encoding) but many others as well.

I believe what most people do is to simply treat ordinal classification as a generic multi-class classification. So, if they have $$K$$ classes, they will have $$K$$ outputs, and simply use cross-entropy as the loss.

But some people have managed to invent a clever encoding for your ordinal classes (see this stackoverflow answer). It's a sort of one-hot encoding,

• class 1 is represented as [0 0 0 0 ...]

• class 2 is represented as [1 0 0 0 ...]

• class 3 is represented as [1 1 0 0 ...]

i.e. each neuron is predicting the probability $$P(\hat y < k)$$. You still have to use a sigmoid as the activation function, but I think this helps the network understanding some continuity between classes, I don't know. Afterwards, you do a post-processing (np.sum) to convert the binary output into your classes.

This strategy resembles the ensemble from Eibe Frank, Mark Hall - A Simple Approach to Ordinal Classification, and I think this is the first publication Jianlin Cheng, Zheng Wang, and Gianluca Pollastri - A neural network approach to ordinal regression of such.

• This approach seems much more appealing. It is important to realize that use predicted modes to turn this into a classification problem is not a good idea. Predicted cumulative probabilities can be turned into predicted individual probabilities, and so the utility function for making a final decision can be inserted much later when utilities are known. See fharrell.com/post/classification . Commented Jan 30, 2018 at 12:29
• @RicardoCruz - Hmm, that sounds a lot like what I had suggested: "1 0 0 for A, 1 1 0 for B, and 1 1 1 for C". Good to know that works! Also wow that was a paper from 2007, this idea has been around for a long time Commented Mar 8, 2018 at 20:51
• Edit to my comment above: Performing "np.sum" on the outputs of the neural network is misleading. The following situation may arise where the output vector is [0 1 0 1 0]. Performing a summation on this vector would produce a class prediction of 2, when in fact the neural network is unsure. Commented Apr 19, 2018 at 15:17
• To perform prediction the paper "A Neural Network Approach to Ordinal Regression" states: "...our methods scans output nodes in the order O1, O2,....,OK. It stop when the output of a node is smaller than the predefined threshold T (e.g. 0.5) or no nodes left. The index k of the last node Ok whose output is bigger than T is the predicted category of the data point." Commented Apr 19, 2018 at 15:30
• @cheesus, I wrote this two years ago and I don't remember... :) I removed that because I also can't make sense of it, thank you for helping improve it! Commented Oct 27, 2020 at 22:27