4
$\begingroup$

I have been working on an ML problem in which I want to predict an interval of money say, a, b, c, d that might be lent to a customer given its credit files, those amounts are represented on ordered bins the i.e a < b < c < d.

First I faced this problem as a multiclass classification problem and even though I did not obtain "good" performance I never thought It could be because of the inherent order on my target.

After googling it I found a paper in which a method to perform classification on ordinal target was developed, but I'm not still sure what are the implications on this scenario and even why it needs special attention.

In that paper is stated:

Standard classification algorithms for nominal classes can be applied to ordinal prediction problems by discarding the ordering information in the class attribute. However, some information is lost when this is done, information that can potentially improve the predictive performance of a classifier.

But it does not clarify the point to me.

Could you please help me to understand the implications of an ordinal target? Might this order, be responsible for poor performance on a multiclass classification task when this order is not considered by applying Logistic Regression, an Ensemble method or any other classification model?

$\endgroup$
5
  • $\begingroup$ Let’s say you want to predict a rating: awful, bad, neutral, good, or excellent. Your prediction is neutral when the true label is good, so you got that wrong. However, you didn’t get it as wrong as if you predicted an awful rating. $\endgroup$ – Dave Oct 23 '20 at 1:09
  • $\begingroup$ So, this is not a matter of the metric you use to evaluate the model rather than the predictions itself? $\endgroup$ – Julio Jesus Luna Oct 23 '20 at 1:25
  • $\begingroup$ I mean, say you define your classes as 0, 1, 2, 3. If you construct a metric such as average((y_hat - y_true)**2) this would penalize more if the true class is for example 0 and you predict 1 vs if you predict 2 or 3. Am I missing something? $\endgroup$ – Julio Jesus Luna Oct 23 '20 at 1:31
  • 1
    $\begingroup$ As the metric tries to optimize, it doesn’t know that it missed the correct classification by just a little. (There’s more to it than that, since we’d be getting probability predictions, but that’s the gist.) $\endgroup$ – Dave Oct 23 '20 at 1:32
  • 1
    $\begingroup$ Those categories could just as easily be coded as $0$, $1$, $2$, $3$, $400$, so treating them as numbers and using square or absolute loss does not quite do what you want to do. $\endgroup$ – Dave Oct 23 '20 at 10:00
3
+50
$\begingroup$

Dave's comments are on the right track. I'll try to expand on them.

Ordinal regression is half-way between classification and real-valued regression. When you perform multiclass classification of your ordinal data, you are assigning the same penalty whenever your classifier predicts a wrong class, no matter which one.

For example, assume that in your problem for some input vector $x$ the right prediction is $a$. Assume you are training two classifiers, $C_1$ and $C_2$. The first one predicts $b$, while the other predicts $d$. In the multivariate classifier's sense, $C_1$ and $C_2$ are equally far off, they have missed the correct class. But from the ordinal regression perspective, $C_1$ is obviously better than $C_2$, since it has missed the correct "class" only by one bin, not by three.

To drive this point into extreme, imagine performing a very-many-classes-classification instead of regression. I.e. you have predictors $x$ and a real-valued response variable $y$. You can treat values of $y$ as classes: $y = 3.14159$ would be one class, $y = 1.4142$ another, and so on. If you had $N$ observations, you're likely to have $N$ different classes (assuming all $y$'s differ). You could try to train a multiclass classifier, but you'd be likely to fail, as there would be only one observation per class. And even if you succeeded (because you were lucky to have same $y$'s repeat multiple times), you'd be essentially having many independent models, where each would only predict its own class and wouldn't care much about the others.

Such an ensemble of models would also be quite complex. If each model has, say, $M$ parameters, and if you had $K$ classes to predict $(K < N)$, your ensemble would have $M \cdot K$ parameters. In contrast, the complexity of the regression model is likely to be independent of the number of distinct $y$ values. You'd settle in advance for a linear, quadratic, or whatever function to fit through your data and the form of the function would determine the number of parameters.

In ordinal regression, e.g. proportional odds logistic regression, it is common to have one set of parameters (a vector) common to all "classes" (i.e. ordinal values), and a set of scalars to distinguish between the individual ordinal values. The same holds also for support vector ordinal regression (see e.g. http://www.gatsby.ucl.ac.uk/~chuwei/paper/svor.pdf), where you have the same model, consisting of the same $\alpha$'s (Lagrange coefficients) for all "classes", and distinguish between the classes only by the corresponding $b$'s (one per "class").

$\endgroup$
2
$\begingroup$

I'd argue that there are two potential complications with discarding the ordering information and just running multiclass regression:

  1. Model complexity: the parameters of the predictions for each category won't be tied together in any way. So the model is trying to learn how to predict category a, b, c, and d as four separate problems, without realizing that there is some structure (e.g. examples of class a will look more similar to those of class b than those of class c or d). This could lead to poor performance on relatively small datasets.

  2. As mentioned in the comments above, the multiclass loss function doesn't take the ordering into account. Presumably in your application mispredicting class a as class b is less bad than mispredicting it as class c.

I can think of a few possible solutions to these issues. One option is to define a similarity matrix across conditions in some way. This could come from some measure of uncertainty between the categories (e.g. a confusion matrix from human labels), or be related to the consequences for misprediction (e.g. if class d is a very elite class that gets a special credit score, that could be designated as less similar to the other classes). This kind of similarity matrix could be used to solve both problems listed above, if it is used for regularization (encouraging similar classes to have similar parameters) and for loss (penalizing mispredictions less harshly for similar classes).

Another possible answer is to just abandon ordinal prediction entirely, and try to predict the amount of money as a continuous value (which you could then discretize into bins if you wanted to). You still might need to think carefully about the loss function (e.g. if these values span multiple orders of magnitude you may want to penalize squared loss of the log of the values, rather than the values themselves).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.