20
$\begingroup$

I have a neural network set up to predict something where the output variable is ordinal. I will describe below using three possible outputs A < B < C.

It is pretty obvious how to use a neural network to output categorical data: the output is just a softmax of the last (usually fully connected) layer, one per category, and the predicted category is the one with the largest output value (this is the default in many popular models). I have been using the same setup for ordinal values. However, in this case the outputs often don't make sense, for example the network outputs for A and C are high but B is low: this is not plausible for ordinal values.

I have one idea for this, which is to calculate loss based on comparing the outputs with 1 0 0 for A, 1 1 0 for B, and 1 1 1 for C. The exact thresholds can be tuned later using another classifier (eg Bayesian) but this seems to capture the essential idea of an ordering of inputs, without prescribing any specific interval scale.

What is the standard way of solving this problem? Is there any research or references that describe the pros and cons of different approaches?

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ I got lots of interesting hits on Google for "ordinal logistic regression" e.g. this paper $\endgroup$ – shadowtalker Mar 3 '15 at 1:53
  • $\begingroup$ @ssdecontrol: Interesting. I tried it; the results were better than picking the one output with the highest value but slightly worse than other methods (naive Bayesian, etc). This is useful, but it doesn't help train the network, only improves results slightly after the fact... or at least I don't see how to make it help train the network. $\endgroup$ – Alex I Mar 3 '15 at 8:22
  • $\begingroup$ which "it" did you try? My only point is that the search engine could be more helpful than you might expect $\endgroup$ – shadowtalker Mar 3 '15 at 8:56
  • $\begingroup$ Also I'm not sure I understand what you mean by "for example the network outputs for A and C are high but B is low: this is not plausible". You mean you're predicting lots of As and Cs but few Bs? I don't see why that should be implausible unless you have substantive or domain-specific reason to think so $\endgroup$ – shadowtalker Mar 3 '15 at 9:05
  • $\begingroup$ I also don't know how you could ever have an output like "1 1 0". I think there's some confusion about terminology here. Are you describing cumulative ordinal outcomes? As in a cumulative logit model? $\endgroup$ – shadowtalker Mar 3 '15 at 9:11
15
$\begingroup$

I believe what most people do is to simply treat ordinal classification as a generic multi-class classification. So, if they have $K$ classes, they will have $K$ outputs, and simply use a sigmoid activation function (not softmax obviously) and binary cross-entropy as the loss.

But some people have managed to invent a clever encoding for your ordinal classes (see this stackoverflow answer). It's a sort of one-hot encoding,

  • class 1 is represented as [0 0 0 0 ...]

  • class 2 is represented as [1 0 0 0 ...]

  • class 3 is represented as [1 1 0 0 ...]

i.e. each neuron is predicting the probability $P(\hat y < k)$. You still have to use a sigmoid as the activation function, but I think this helps the network understanding some continuity between classes, I don't know. Afterwards, you do a post-processing (np.sum) to convert the binary output into your classes.

This strategy resembles the ensemble from Frank and Hall, and I think this is the first publication of such.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ This approach seems much more appealing. It is important to realize that use predicted modes to turn this into a classification problem is not a good idea. Predicted cumulative probabilities can be turned into predicted individual probabilities, and so the utility function for making a final decision can be inserted much later when utilities are known. See fharrell.com/post/classification . $\endgroup$ – Frank Harrell Jan 30 '18 at 12:29
  • 1
    $\begingroup$ @RicardoCruz - Hmm, that sounds a lot like what I had suggested: "1 0 0 for A, 1 1 0 for B, and 1 1 1 for C". Good to know that works! Also wow that was a paper from 2007, this idea has been around for a long time $\endgroup$ – Alex I Mar 8 '18 at 20:51
  • $\begingroup$ Yeah, I was surprised myself when I found that paper! $\endgroup$ – Ricardo Cruz Mar 8 '18 at 23:15
  • $\begingroup$ Note: As stated in "A Neurel Network Approach to Ordinal Regression": "...using independent sigmoid functions for output nodes does not guaranttee the monotonic relation (o1 >= o2 >= .... >= oK), which is not necessary but, desirable for making predictions." Therefore, just performing an "np.sum" at prediction time is not the best method. $\endgroup$ – Scott Apr 19 '18 at 14:51
  • 1
    $\begingroup$ Edit to my comment above: Performing "np.sum" on the outputs of the neural network is misleading. The following situation may arise where the output vector is [0 1 0 1 0]. Performing a summation on this vector would produce a class prediction of 2, when in fact the neural network is unsure. $\endgroup$ – Scott Apr 19 '18 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.