Timeline for When a classifier predicting probability should be calibrated?
Current License: CC BY-SA 4.0
20 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 11, 2021 at 12:46 | vote | accept | Sergey Bushmanov | ||
| Jan 15, 2021 at 11:55 | answer | added | Sergey Bushmanov | timeline score: 6 | |
| Jan 15, 2021 at 6:55 | history | edited | Sergey Bushmanov | CC BY-SA 4.0 |
added 42 characters in body
|
| Jan 13, 2021 at 9:40 | history | edited | Sergey Bushmanov | CC BY-SA 4.0 |
added 43 characters in body
|
| Jan 13, 2021 at 3:00 | history | tweeted | twitter.com/StackStats/status/1349189543539437571 | ||
| Jan 12, 2021 at 22:53 | comment | added | seanv507 | catboost etc are typically trained to optimise log loss - are you saying that's the wrong objective function?for RF choosing hyperparameters by accuracy is wrong, see arxiv.org/pdf/1812.05792.pdf eg fig 11, where it is shown that using the defaults for RF regression give accurate probability estimates | |
| Jan 12, 2021 at 21:36 | comment | added | Ben Reiniger | sklearn random forests do soft voting, i.e. averages the probabilities across trees. See the last sentence of scikit-learn.org/stable/modules/ensemble.html#random-forests | |
| Jan 12, 2021 at 21:16 | comment | added | Sergey Bushmanov |
@seanv507 RF/catboost etc because the wrong parameters are used (eg leaf nodes having a single sample) I believe calibration is needed not to overcome memorizing of data due to model params misspecification but to overcome misspecification of objective function. We are talking about well performing classifiers for one task, but having bias in predicting probabilities
|
|
| Jan 12, 2021 at 21:00 | comment | added | seanv507 | agree with @Tim. This is 5 questions. each model could be uncalibrated for a different reason. a) a 'linear' logistic regression - because you need non linearities in your inputs.b) RF/catboost etc because the wrong parameters are used (eg leaf nodes having a single sample) c) naive bayes - because it is not a probabilistic classifier 9unless features are independent) | |
| Jan 12, 2021 at 20:55 | history | edited | Sergey Bushmanov | CC BY-SA 4.0 |
added 46 characters in body
|
| Jan 12, 2021 at 20:44 | history | edited | Sergey Bushmanov | CC BY-SA 4.0 |
added 46 characters in body
|
| Jan 12, 2021 at 20:33 | history | edited | Sergey Bushmanov | CC BY-SA 4.0 |
deleted 592 characters in body
|
| Jan 12, 2021 at 20:30 | comment | added | Sergey Bushmanov | @Tim. Thanks, expected. Will edit. | |
| Jan 12, 2021 at 20:29 | comment | added | Tim | If I may suggest something, it'd be probably better if you could make the question more focused, since you are about five things at once. Questions that are overtly broad are less likely to be answered from my experience, better ask few separate questions than all-in-one. | |
| Jan 12, 2021 at 20:28 | history | edited | Sergey Bushmanov | CC BY-SA 4.0 |
added 69 characters in body
|
| Jan 12, 2021 at 20:26 | comment | added | Sergey Bushmanov |
@Tim Thanks for pointing that out. Unfortunately, the accepted answer states [RF] ... will end up with a better-calibrated model in practice than a logistic regression model, which is contrary to what is advised by sklearn, in the literature, and seen in the simulation. A comment by @whuber scikit-learn quotation about "optimizes log loss" isn't an effective explanation, because there is no necessary connection between this and being unbiased is more interesting, and I'd appreciate if somebody can elaborate on that.
|
|
| Jan 12, 2021 at 20:06 | comment | added | Tim | The first question is answered in: stats.stackexchange.com/questions/390487/… | |
| Jan 12, 2021 at 19:49 | history | edited | Sergey Bushmanov | CC BY-SA 4.0 |
edited tags
|
| Jan 12, 2021 at 17:07 | history | edited | Sergey Bushmanov | CC BY-SA 4.0 |
edited body
|
| Jan 12, 2021 at 16:59 | history | asked | Sergey Bushmanov | CC BY-SA 4.0 |