34
$\begingroup$

I have a neural network set up to predict something where the output variable is ordinal. I will describe below using three possible outputs A < B < C.

It is pretty obvious how to use a neural network to output categorical data: the output is just a softmax of the last (usually fully connected) layer, one per category, and the predicted category is the one with the largest output value (this is the default in many popular models). I have been using the same setup for ordinal values. However, in this case the outputs often don't make sense, for example the network outputs for A and C are high but B is low: this is not plausible for ordinal values.

I have one idea for this, which is to calculate loss based on comparing the outputs with 1 0 0 for A, 1 1 0 for B, and 1 1 1 for C. The exact thresholds can be tuned later using another classifier (eg Bayesian) but this seems to capture the essential idea of an ordering of inputs, without prescribing any specific interval scale.

What is the standard way of solving this problem? Is there any research or references that describe the pros and cons of different approaches?

$\endgroup$
9
  • 1
    $\begingroup$ I got lots of interesting hits on Google for "ordinal logistic regression" e.g. this paper $\endgroup$ Mar 3, 2015 at 1:53
  • $\begingroup$ @ssdecontrol: Interesting. I tried it; the results were better than picking the one output with the highest value but slightly worse than other methods (naive Bayesian, etc). This is useful, but it doesn't help train the network, only improves results slightly after the fact... or at least I don't see how to make it help train the network. $\endgroup$
    – Alex I
    Mar 3, 2015 at 8:22
  • $\begingroup$ which "it" did you try? My only point is that the search engine could be more helpful than you might expect $\endgroup$ Mar 3, 2015 at 8:56
  • $\begingroup$ Also I'm not sure I understand what you mean by "for example the network outputs for A and C are high but B is low: this is not plausible". You mean you're predicting lots of As and Cs but few Bs? I don't see why that should be implausible unless you have substantive or domain-specific reason to think so $\endgroup$ Mar 3, 2015 at 9:05
  • 1

2 Answers 2

25
$\begingroup$

I believe what most people do is to simply treat ordinal classification as a generic multi-class classification. So, if they have $K$ classes, they will have $K$ outputs, and simply use cross-entropy as the loss.

But some people have managed to invent a clever encoding for your ordinal classes (see this stackoverflow answer). It's a sort of one-hot encoding,

  • class 1 is represented as [0 0 0 0 ...]

  • class 2 is represented as [1 0 0 0 ...]

  • class 3 is represented as [1 1 0 0 ...]

i.e. each neuron is predicting the probability $P(\hat y < k)$. You still have to use a sigmoid as the activation function, but I think this helps the network understanding some continuity between classes, I don't know. Afterwards, you do a post-processing (np.sum) to convert the binary output into your classes.

This strategy resembles the ensemble from Frank and Hall, and I think this is the first publication of such.

$\endgroup$
9
  • $\begingroup$ This approach seems much more appealing. It is important to realize that use predicted modes to turn this into a classification problem is not a good idea. Predicted cumulative probabilities can be turned into predicted individual probabilities, and so the utility function for making a final decision can be inserted much later when utilities are known. See fharrell.com/post/classification . $\endgroup$ Jan 30, 2018 at 12:29
  • 1
    $\begingroup$ @RicardoCruz - Hmm, that sounds a lot like what I had suggested: "1 0 0 for A, 1 1 0 for B, and 1 1 1 for C". Good to know that works! Also wow that was a paper from 2007, this idea has been around for a long time $\endgroup$
    – Alex I
    Mar 8, 2018 at 20:51
  • 4
    $\begingroup$ Edit to my comment above: Performing "np.sum" on the outputs of the neural network is misleading. The following situation may arise where the output vector is [0 1 0 1 0]. Performing a summation on this vector would produce a class prediction of 2, when in fact the neural network is unsure. $\endgroup$
    – sccrthlt
    Apr 19, 2018 at 15:17
  • 1
    $\begingroup$ To perform prediction the paper "A Neural Network Approach to Ordinal Regression" states: "...our methods scans output nodes in the order O1, O2,....,OK. It stop when the output of a node is smaller than the predefined threshold T (e.g. 0.5) or no nodes left. The index k of the last node Ok whose output is bigger than T is the predicted category of the data point." $\endgroup$
    – sccrthlt
    Apr 19, 2018 at 15:30
  • 1
    $\begingroup$ @cheesus, I wrote this two years ago and I don't remember... :) I removed that because I also can't make sense of it, thank you for helping improve it! $\endgroup$ Oct 27, 2020 at 22:27
9
$\begingroup$

I think the approach to only encode the ordinal labels as

  • class 1 is represented as [0 0 0 0 ...]

  • class 2 is represented as [1 0 0 0 ...]

  • class 3 is represented as [1 1 0 0 ...]

and use binary cross-entropy as the loss function is suboptimal. As mentioned in the comments, it might happen that the predicted vector is for example [1 0 1 0 ...]. This is undesirable for making predictions.

The paper Rank-consistent ordinal regression for neural networks describes how to restrict the neural network to make rank-consistent predictions. You have to make sure that the last layer shares its weights, but should have different biases. You can implement this in Tensorflow by adding the following as the last part of the network (credits for https://stackoverflow.com/questions/59656313/how-to-share-weights-and-not-biases-in-keras-dense-layers):

class BiasLayer(tf.keras.layers.Layer):
    def __init__(self, units, *args, **kwargs):
        super(BiasLayer, self).__init__(*args, **kwargs)
        self.bias = self.add_weight('bias',
                                    shape=[units],
                                    initializer='zeros',
                                    trainable=True)

    def call(self, x):
        return x + self.bias


# Add the following as the output of the Sequential model
model.add(keras.layers.Dense(1, use_bias=False))
model.add(BiasLayer(4))
model.add(keras.layers.Activation("sigmoid"))

Note that the number of ordinal classes here is 5, hence the $K-1$ biases.

I tested the difference in performance on actual data, and the predictive accuracy improved substantially. Hope this helps.

$\endgroup$
5
  • $\begingroup$ Neat!! I am searching for the resource to implement the original regression in PyTorch. "I tested the difference in performance on actual data and the predictive accuracy improved substantially." Do you mean the accuracy on the testing set? Which metric are you using? which models did you compare it with? $\endgroup$
    – Phume
    Nov 8, 2020 at 3:44
  • 2
    $\begingroup$ I highly recommend this answer, I have been able to successfully implement without much additional work because of this repo: github.com/ck37/coral-ordinal $\endgroup$
    – Chris Farr
    Dec 16, 2020 at 17:13
  • $\begingroup$ If I understand the paper, then the Keras-based solution in the answer by @thijsvpd also guarantees the proper ordering of the biases, right? If so, this should be the top answer! $\endgroup$ Jul 29, 2021 at 18:56
  • $\begingroup$ @thijsvdp Have you encountered ordinal function with two-side ranking? For example, ordinal classes are [0, 1, 2, 3, 4] and π‘ƒπ‘Žπ‘Ÿπ‘”_π‘šπ‘Žπ‘₯=2, then it should be 𝑃(0)<𝑃(1)<𝑃(2)>𝑃(3)>𝑃(4) $\endgroup$ Sep 29, 2021 at 10:14
  • $\begingroup$ I guess this is not possible, because if I understand you correctly you have the following p0<p1<p2 and p4<p3<p2. This lacks information about the ordering of p0, p1, p3 and p4. For example, it can happen that p0<p4 or p4<p0. That information is not provided. I guess you can build a model which models both rankings separately. But if you want a single class prediction that cannot be done here because of lacking information between the ordering of p0 and p1 versus p3 and p4. $\endgroup$
    – thijsvdp
    Sep 29, 2021 at 11:41

Your Answer

By clicking β€œPost Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.