5
$\begingroup$

Assume we have a classification problem with two classes $\{-1,1\}$. In my research I rather need probability, than just predicted classes. I use isotonic regression to calibrate classifier. In sklearn in the description it says:

Methods such as bagging and random forests that average predictions from a base set of models can have difficulty making predictions near 0 and 1 because variance in the underlying base models will bias predictions that should be near zero or one away from these values.

Is my following conclusion correct?

Conclusion: If one uses random forest then one should not count to much on the estimators of probability which is close to either $0$ or $1$. Though, if the estimator is somewhere close to $0.5$, then the estimator is more informative. Also, assume that in my case this classifier is the first step, for example, in stacked model. Then, there is no much sense in choosing very high threshold, cause the estimator is not very accurate.

enter image description here

$\endgroup$
2
  • 1
    $\begingroup$ Is there some reason why you're not using the almost perfectly calibrated logistic model? Also, I don't quite get your interpretation of the calibration curves: the other models don't seem too bad at the very far limits, but do poorly at intermediate values (probabilities between 0.2 to 0.4, or between 0.6 and 0.8, depending on the model). $\endgroup$
    – EdM
    May 20, 2021 at 15:40
  • $\begingroup$ Dear @EdM, the figures I sent are taken from sklearn. It is just for visualisation. I use random forest, cause it is the best for my research. $\endgroup$
    – ABK
    May 20, 2021 at 16:02

1 Answer 1

6
+50
$\begingroup$

The first part of your conclusion is correct. Sklearn computes probabilities in the RFC by classifying subsamples and then averaging the proportion of observations in each class. This will bias probabilities that should be near 0 and 1 inward. This is simply because for a class to have a probability near 0 or 1 we need that many of the subsample classifiers also tend to classify all or none of the observations within that class. You should read this paper that goes into some more details and compares different methods for retrieving probabilities: https://www.cs.cornell.edu/~alexn/papers/calibration.icml05.crc.rev3.pdf

For the second part of your question, how much you should care about this all depends on what you are doing next, why you need these probabilities, and what the whole model looks like. In general, it's not a great idea to used biased estimates of a conditional probability in a second stage. If you are considering trimming along a cutoff this can also be problematic as this shifts the observed distribution of the probabilities.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.