# How to set up neural network to output ordinal data?

I have a neural network set up to predict something where the output variable is ordinal. I will describe below using three possible outputs A < B < C.

It is pretty obvious how to use a neural network to output categorical data: the output is just a softmax of the last (usually fully connected) layer, one per category, and the predicted category is the one with the largest output value (this is the default in many popular models). I have been using the same setup for ordinal values. However, in this case the outputs often don't make sense, for example the network outputs for A and C are high but B is low: this is not plausible for ordinal values.

I have one idea for this, which is to calculate loss based on comparing the outputs with 1 0 0 for A, 1 1 0 for B, and 1 1 1 for C. The exact thresholds can be tuned later using another classifier (eg Bayesian) but this seems to capture the essential idea of an ordering of inputs, without prescribing any specific interval scale.

What is the standard way of solving this problem? Is there any research or references that describe the pros and cons of different approaches?

• I got lots of interesting hits on Google for "ordinal logistic regression" e.g. this paper Mar 3, 2015 at 1:53
• @ssdecontrol: Interesting. I tried it; the results were better than picking the one output with the highest value but slightly worse than other methods (naive Bayesian, etc). This is useful, but it doesn't help train the network, only improves results slightly after the fact... or at least I don't see how to make it help train the network. Mar 3, 2015 at 8:22
• which "it" did you try? My only point is that the search engine could be more helpful than you might expect Mar 3, 2015 at 8:56
• Also I'm not sure I understand what you mean by "for example the network outputs for A and C are high but B is low: this is not plausible". You mean you're predicting lots of As and Cs but few Bs? I don't see why that should be implausible unless you have substantive or domain-specific reason to think so Mar 3, 2015 at 9:05
• Aug 1, 2018 at 21:49

I believe what most people do is to simply treat ordinal classification as a generic multi-class classification. So, if they have $$K$$ classes, they will have $$K$$ outputs, and simply use cross-entropy as the loss.

But some people have managed to invent a clever encoding for your ordinal classes (see this stackoverflow answer). It's a sort of one-hot encoding,

• class 1 is represented as [0 0 0 0 ...]

• class 2 is represented as [1 0 0 0 ...]

• class 3 is represented as [1 1 0 0 ...]

i.e. each neuron is predicting the probability $$P(\hat y < k)$$. You still have to use a sigmoid as the activation function, but I think this helps the network understanding some continuity between classes, I don't know. Afterwards, you do a post-processing (np.sum) to convert the binary output into your classes.

This strategy resembles the ensemble from Frank and Hall, and I think this is the first publication of such.

• This approach seems much more appealing. It is important to realize that use predicted modes to turn this into a classification problem is not a good idea. Predicted cumulative probabilities can be turned into predicted individual probabilities, and so the utility function for making a final decision can be inserted much later when utilities are known. See fharrell.com/post/classification . Jan 30, 2018 at 12:29
• @RicardoCruz - Hmm, that sounds a lot like what I had suggested: "1 0 0 for A, 1 1 0 for B, and 1 1 1 for C". Good to know that works! Also wow that was a paper from 2007, this idea has been around for a long time Mar 8, 2018 at 20:51
• Edit to my comment above: Performing "np.sum" on the outputs of the neural network is misleading. The following situation may arise where the output vector is [0 1 0 1 0]. Performing a summation on this vector would produce a class prediction of 2, when in fact the neural network is unsure. Apr 19, 2018 at 15:17
• To perform prediction the paper "A Neural Network Approach to Ordinal Regression" states: "...our methods scans output nodes in the order O1, O2,....,OK. It stop when the output of a node is smaller than the predefined threshold T (e.g. 0.5) or no nodes left. The index k of the last node Ok whose output is bigger than T is the predicted category of the data point." Apr 19, 2018 at 15:30
• @cheesus, I wrote this two years ago and I don't remember... :) I removed that because I also can't make sense of it, thank you for helping improve it! Oct 27, 2020 at 22:27

I think the approach to only encode the ordinal labels as

• class 1 is represented as [0 0 0 0 ...]

• class 2 is represented as [1 0 0 0 ...]

• class 3 is represented as [1 1 0 0 ...]

and use binary cross-entropy as the loss function is suboptimal. As mentioned in the comments, it might happen that the predicted vector is for example [1 0 1 0 ...]. This is undesirable for making predictions.

The paper Rank-consistent ordinal regression for neural networks describes how to restrict the neural network to make rank-consistent predictions. You have to make sure that the last layer shares its weights, but should have different biases. You can implement this in Tensorflow by adding the following as the last part of the network (credits for https://stackoverflow.com/questions/59656313/how-to-share-weights-and-not-biases-in-keras-dense-layers):

class BiasLayer(tf.keras.layers.Layer):
def __init__(self, units, *args, **kwargs):
super(BiasLayer, self).__init__(*args, **kwargs)
shape=[units],
initializer='zeros',
trainable=True)

def call(self, x):
return x + self.bias

# Add the following as the output of the Sequential model

Note that the number of ordinal classes here is 5, hence the $$K-1$$ biases.