Timeline for Custom metrics for multiclass classification when class errors have different weights
Current License: CC BY-SA 4.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 13, 2020 at 12:58 | comment | added | Stephan Kolassa | @RichardHardy: you won't be surprised to hear that I still disagree with your position, although I do agree that the thread makes for thought-provoking reading. | |
| Apr 13, 2020 at 8:25 | comment | added | Richard Hardy | One of my favorite threads is "When is it appropriate to use an improper scoring rule?". Both my comment there and Cagdas Ozgenc's answer show that when probabilities are not the goal in themselves, proper scoring rules can be harmful. Now, it is usually the case that people are interested in making decisions, not learning probabillities for their own sake. I believe the OPs case is one of them, too. | |
| Apr 9, 2020 at 9:29 | comment | added | Stephan Kolassa | @RichardHardy: Hm. We may indeed be talking past each other. I agree that I am only addressing how to fit a probabilistic model. Making decisions based on that model (and costs of mis-decisions) is a separate step. IMO, the common conflation of the two steps is misleading, to say the least. I should probably have linked to my perennial favorite for how to get from a probabilistic prediction to an action. | |
| Apr 9, 2020 at 9:09 | comment | added | Richard Hardy | We are probably talking about different loss functions. Following your suggestion, there would be two loss functions: one for fitting the probabilities, another for making decisions using the fitted probabilities. Meanwhile, the OP was probably using a setup with a single loss function that played both roles at the same time. My comments tried to address this, as I think you neglected the second loss function without which no decisions are possible. I am still curious about your answer to the very first question of mine, even if the answer is "I don't know". | |
| Apr 9, 2020 at 6:53 | history | edited | Stephan Kolassa | CC BY-SA 4.0 |
added 471 characters in body
|
| Apr 9, 2020 at 6:50 | comment | added | Stephan Kolassa | @RichardHardy: Well, the OP asked how to tune the loss function used in their model fitting, and I would propose using a proper scoring rule. Let me edit that into the post... | |
| Apr 9, 2020 at 6:13 | comment | added | Richard Hardy | I do not think the answer is short, no problem there. However, I think it is incomplete. What is your advice for completing the task at hand, the task being not to evaluate predictive probabilities but to optimize a classification algorithm? This brings me back to my first comment, and I am curious to learn you opinion. | |
| Apr 9, 2020 at 5:50 | comment | added | Stephan Kolassa | @RichardHardy: I agree that this is a very short answer. I would say it's better to have a short answer than no answer at all. Anyone who has a better answer can post it. | |
| Apr 9, 2020 at 5:49 | comment | added | Stephan Kolassa | Well, the point of proper scoring rules is precisely that they will be minimized (or maximized, depending on the formulation) if your predicted possibilities are the true ones. If you work with outputs that measure something different, then you only have a proxy, and you don't know whether optimizing your proper scoring rule will not lead you astray, towards biased predictions. But it will probably be better than trying to optimize accuracy. | |
| Apr 8, 2020 at 21:47 | comment | added | A1010 | Thank you for the link at your ansewer @Stephan. Is it strinctly mandatory that the output of the classification algorithms is a probability or could it be just a score? Obviously the higher the score the higher the confidence for the selected class, but, for example, the sum of all the scores could be different from 1. Do you believe it could be a problem for the application of a scoring function? | |
| Apr 8, 2020 at 16:03 | comment | added | Richard Hardy | So if the OP is currently using XGBoost and random forest, would you suggest training them to predict probabilities rather than class labels and then using the predicted probabilities for classification based on, say, minimization of expected loss? (Also, your answer stops short of telling how to actually do classification based on estimated probabilities. But that of course is simple if one just aims at minimizing the expected loss.) | |
| Apr 8, 2020 at 15:49 | history | answered | Stephan Kolassa | CC BY-SA 4.0 |