3
$\begingroup$

I am working on getting good probability from Random Forest algorithm for better decision making. Currently, I have trained the RF model with default parameters and then applied isotonic regression using CaliberatedCV sklearn library. Although I was able to get some improvement in terms of log loss, the brier score has not changed. I would like to request to help me find how I can get nearly perfect probability calibration for a multiclass problem from my Random Forest algorithm.

The dataset is highly imbalanced, therefore, I have set class weight parameter of RF to "balanced".

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Hi Tazar, can you please explain more about the larger picture here. What are the data exactly? What goal are you trying to reach with this model? $\endgroup$
    – Robert Dodier
    Commented Nov 10, 2020 at 17:14
  • $\begingroup$ Robert Dodier its a flow based intrusion detection dataset, My goal is to understand when my ML algorithm predicat an instance as attack or normal. I would be interested to see how much confidence is associated with its prediction so that these confidence can be used further for any decision making $\endgroup$
    – Tazar Hussain
    Commented Nov 10, 2020 at 17:36
  • 1
    $\begingroup$ Some related discussion stats.stackexchange.com/search?q=calibrate+random+forest $\endgroup$
    – Sycorax ♦
    Commented Nov 11, 2020 at 17:05

1 Answer 1

Reset to default
2
$\begingroup$

I'll bring over my comments from the Stackoverflow question.

(1) RF outputs are votes, not proportions. My advice is to try to set parameters for RF or just use a different kind of tree or different model altogether to get at least proportion of positive examples in each leaf. Starting from votes is just setting your starting position farther away from the goal.

(2) Since the cases are highly imbalanced, consider constructing generative models, one for positive, one for negative cases, then combine them via Bayes rule (which brings the base rates into play). If there is anything you know specifically about the minor class, try to build that into the generative model.

Whatever you know about how the data are generated is very relevant, especially in the case of building a generative model. So any further information about that will help others help you.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.