Timeline for Calibrating the probabilities of Ridge Classifier on imbalanced dataset
Current License: CC BY-SA 4.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 1, 2022 at 18:40 | vote | accept | Octave | ||
| Oct 1, 2022 at 18:40 | comment | added | Octave | Thanks for the clarification @BenReiniger this makes a lot more sense now! | |
| Oct 1, 2022 at 15:46 | comment | added | Ben Reiniger | I guess there is more to say about just the probabilities though. It's probably correct to say that no individual has a 50% chance of defaulting. That would require a really strong indicator of bad credit (or fraud). The main reason of defaults (in prime customers) are random external costs that would be very hard to predict at time of application. | |
| Oct 1, 2022 at 15:44 | comment | added | Ben Reiniger | @Octave I feel like you're focusing on the two probabilities too much. Ignore the probability of non-default, it's just the complement to the probability of default. And maybe think less about the classification meaning "this person will (likely) default" and more about "I shouldn't lend to this person, I will (on average) lose money". The latter requires a bit more analysis (what's expected loss/profit if they default/not?), but after that the threshold you choose may be quite small. (Real business is more nuanced, but this is a good start to understanding the imbalance in decisioning.) | |
| Sep 30, 2022 at 12:52 | comment | added | Octave | Now when I calibrate the probabilities all of them are above 0.5 for the majority class and below 0.5 for the minority class. How am I supposed to consume this distribution? It just seems illogical to me and I feel like these probabilities are worthless because I wouldn’t be able to predict the class based on them. | |
| Sep 30, 2022 at 12:47 | comment | added | Octave | Thanks a lot for your very insightful answers and I apoligize for being dense but here is what I don’t understand : Ridge Regression takes 0-1 as an input, converts them in -1/1, applies regression and predicts classes based on the sign of the results. Now when I apply the soft max to the decision function it returns “logical” probabilities, IE when the probability is > 0.5 it predicts the class. | |
| Sep 30, 2022 at 12:38 | comment | added | Ben Reiniger | @Octave Ridge sets a threshold at zero (the regression being fitted with targets $\pm1$); threshold analysis is generally left up to you in sklearn. | |
| Sep 29, 2022 at 19:31 | comment | added | Octave | In that case, the Ridge Classifying algorithm automatically determines the threshold? Is it possible to know what threshold it selects? | |
| Sep 29, 2022 at 18:04 | comment | added | Ben Reiniger | @Octave You need to decide how to consume the predicted probabilities. For example, you might compute the expected profit of the loan and approve positive-expected-value (or maybe some minimum positive threshold profitability) applications, which may translate to a probability threshold of say 5%. | |
| Sep 29, 2022 at 17:56 | comment | added | Octave | I understand but then I have a hard time figuring out how the pipeline actually predicts the minority class when all samples have a > 50% probability of belonging to the majority class. | |
| Sep 29, 2022 at 16:37 | comment | added | Ben Reiniger | The point is that calibration is giving you better probability estimates, and that those are naturally skewed low. You shouldn't pay much attention to any results relying on a 50% cutoff; your lender probably won't want to lend money to someone with a 49% probability of default! | |
| Sep 29, 2022 at 15:38 | comment | added | Octave | Thanks a lot for your thoughtful answrer. I have made significant edits in my main post and added a Kaggle link as requested. What I meant with the calibrated probabilities histogram is that there is no sample with a probability of belonging to the minority class > 0.5. So unless I'm completely misunderstanding what it represents I think it would mean that the calibrated model predicts that all samples belong to the minory class. I have provided the probability histograms as an edit also. It's not impossible that I'm completely misrepresenting the histograms though, I'm still learning! | |
| Sep 29, 2022 at 14:23 | history | answered | Ben Reiniger | CC BY-SA 4.0 |