2
$\begingroup$

I have a multiclass classification problem (eg. the target variable is made by 4 different outcomes: Product A, Product B, Product C and NO Product). Not all the errors are equal: for example, if the true label is "Product A" and the prediction is "NO Product" it is not a big problem, while if the true label is "Product C" the impact of the error is much bigger. Basically, I have to insert this information into the loss function of the algorithm (I am currently using Xg-Boost, Random Forest, etc).

I know that it's possible, for example using scikit-learn to train algorithms using the parameter class_weight in a way that a specific class error is more important.

Is there the possibility to say, for example, that a miss-classification of product C in Product B is more problematic w.r.t a miss-classification of Product C to Product B? I am basically looking at something that penalize the model for making cross predictions.

Any idea? Thank you!

Improve this question
$\endgroup$
2

1 Answer 1

Reset to default
1
$\begingroup$

Use a method that outputs predictive probabilities, as:

My prediction is a probability 0.6 that this instance is product A, 0.2 for product B, 0.15 for product C and 0.05 for NO product.

Then use a proper scoring rule to evaluate these predictive probabilities. If you can tune the objective function you supply to your model fitting algorithm, you should be able to use such a proper scoring rule. (The tag may be helpful.)

Of course, all the usual caveats about overfitting apply. It would be best to do this in a cross-validation setup. Also, note whether the proper scoring rule is "positively or negatively oriented", i.e., whether larger or smaller is better - different people use different conventions.

You can find more information at Why is accuracy not the best measure for assessing classification models?, which refers to two-class classification, but my answer there applies to multiclass cases, as well.

Improve this answer
$\endgroup$
10
  • $\begingroup$ So if the OP is currently using XGBoost and random forest, would you suggest training them to predict probabilities rather than class labels and then using the predicted probabilities for classification based on, say, minimization of expected loss? (Also, your answer stops short of telling how to actually do classification based on estimated probabilities. But that of course is simple if one just aims at minimizing the expected loss.) $\endgroup$
    – Richard Hardy
    Commented Apr 8, 2020 at 16:03
  • $\begingroup$ Thank you for the link at your ansewer @Stephan. Is it strinctly mandatory that the output of the classification algorithms is a probability or could it be just a score? Obviously the higher the score the higher the confidence for the selected class, but, for example, the sum of all the scores could be different from 1. Do you believe it could be a problem for the application of a scoring function? $\endgroup$
    – A1010
    Commented Apr 8, 2020 at 21:47
  • $\begingroup$ Well, the point of proper scoring rules is precisely that they will be minimized (or maximized, depending on the formulation) if your predicted possibilities are the true ones. If you work with outputs that measure something different, then you only have a proxy, and you don't know whether optimizing your proper scoring rule will not lead you astray, towards biased predictions. But it will probably be better than trying to optimize accuracy. $\endgroup$
    – Stephan Kolassa
    Commented Apr 9, 2020 at 5:49
  • $\begingroup$ @RichardHardy: I agree that this is a very short answer. I would say it's better to have a short answer than no answer at all. Anyone who has a better answer can post it. $\endgroup$
    – Stephan Kolassa
    Commented Apr 9, 2020 at 5:50
  • $\begingroup$ I do not think the answer is short, no problem there. However, I think it is incomplete. What is your advice for completing the task at hand, the task being not to evaluate predictive probabilities but to optimize a classification algorithm? This brings me back to my first comment, and I am curious to learn you opinion. $\endgroup$
    – Richard Hardy
    Commented Apr 9, 2020 at 6:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.