11
$\begingroup$

I am working on random forest classifiation with a dataset size of 977 records and 6 features. However, my class is imbalanced and proportion is 77:23

I was reading about calibration of models (binary classification) to improve/calibrate the predicted probabilities of actually fitted model (RF in this case).

However, I also found out that calibration model has to be fit using a different dataset.

But the problem is, I already used sklearn train and test split - 680 records for my train and 297 records for my test (of random forest model)

Now, how can I calibrate my model (as I don't have any new data)

Especially, as I am using Random forest, I wish to calibrate my model for better predicted probabilities?

If you ae interested to look at my calibration curve and brier score loss, please find below

enter image description here

update - extra trees classifier

enter image description here

update - logistic regression

enter image description here

update - bootstrap optimisim

enter image description here

$\endgroup$
26
  • 4
    $\begingroup$ One solution: use a model that doesn't have such a problem, like logistic regression. $\endgroup$
    – Tim
    Apr 2, 2022 at 10:49
  • $\begingroup$ @Tim - My data shows some non-linear behavior. With logistic regression, the performance is even poor on train data $\endgroup$
    – The Great
    Apr 2, 2022 at 10:50
  • 1
    $\begingroup$ @TheGreat you could try Kernel Logistic Regression (KLR) which can deal very well with non-linear behaviour or Gassian Process Classifiers. For calibrating the Random Forest though, you could use the "out-of-bag" output (the output formed by the ensemble of all of the trees that didn't have a particular training pattern in their training set). $\endgroup$ Apr 2, 2022 at 11:59
  • $\begingroup$ @DikranMarsupial - Is there any tutorial or llearning resurces that you can refer me to for random forest calibration based on out of bag error? $\endgroup$
    – The Great
    Apr 2, 2022 at 12:04
  • $\begingroup$ @TheGreat not that I know of, I've not used random forest much, but I do use bagging for other purposes, where I have used the out-of-bag estimate for model selection. In general though, if you really want the probabilities, you are better off using an algorithm that estimates them directly rather than using a more discrete classifier and re-calibrating. The raw score of the RF isn't necessarily a good basis for estimating probabilities. $\endgroup$ Apr 2, 2022 at 12:12

1 Answer 1

10
$\begingroup$

I also found out that calibration model has to be fit using a different dataset.

That's not strictly true. As Frank Harrell explains, with data sets of this size it's generally best to develop the model on the entire data set and then validate the modeling process by repeating the modeling on multiple bootstrap samples and evaluating performance on the full data set. (Repeated cross validation, as suggested by usεr11852, can also work for this.) That allows evaluation of and correction for bias, and production of calibration curves that are likely to represent the quality of the model when applied to new data samples from the population. This presentation outlines the procedure in the context of logistic regression, but the principles are general.

$\endgroup$
18
  • $\begingroup$ thanks for your help. upvoted. While I understand that my dataset size is small, but when you suggest to develop the model on full dataset and later validate on same datapoints (stored as bootstrap samples)/full dataset, isn't that called as data leakage? I was of the understanding (not an expert) that whatever data that we used for train, test and validation, should be different (and they should not be exposed or overalapping with data points from other model building phases) $\endgroup$
    – The Great
    Apr 2, 2022 at 14:25
  • $\begingroup$ @TheGreat the argument for bootstrap validation is that the process of taking bootstrap samples from your data mimics the process of taking your data from the underlying population. Completely separating train/test/validate sets evaluates the model itself, but it leads to problems like yours with data sets smaller than a few tens of thousands. So you evaluate the modeling process with this well-established approach. That's not the same as evaluating the final model, but it shows how well your approach is expected to work. $\endgroup$
    – EdM
    Apr 2, 2022 at 14:37
  • 2
    $\begingroup$ The reason that the Efron-Gong optimism bootstrap does not involve "data leakage" is that it estimates the difference between a super-overfitted model (one fit on a bootstrap sample) to a regular-overfitted model (evaluate on the original dataset). The difference between super-overfitting and regular overfitting = difference between regular overfitting and no overfitting, hence the estimated optimism is the amount of overfitting. $\endgroup$ Apr 2, 2022 at 14:39
  • 1
    $\begingroup$ @TheGreat there is so much developed for R for this that it's hard to get motivated to write in python. $\endgroup$ Apr 2, 2022 at 15:57
  • 3
    $\begingroup$ @TheGreat it's a way to anticipate problems, not to check them. Look at Harrell's course notes and book, particularly Chapter 4. Those are in the context of regression modeling, but the guidance is useful in general. In searching for "random forest" in those documents, I saw that random forests might require a higher ratio than the 20/1 rule of thumb in logistic regression to avoid overfitting. So my prior comment might have been over-optimistic. $\endgroup$
    – EdM
    Apr 2, 2022 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.