3
$\begingroup$

I have trained a random forest classifier on a (highly-imbalanced) 3-class problem (A 1% of the data, B 96%, C 3%) and obtained probabilities for each of the three classes.

Currently I assign an observation to the class with the highest probability.

Is there a possibility to adjust the thresholds so I can make a statement like "With maximal 1% probability we will be predicting B though it is actually A"? How can this be done? I really need to control for this type of error while other errors are not that important.

$\endgroup$
5
  • $\begingroup$ @StephanKolassa I don't think this is a duplicate. The OP wants to know how to shift the classification decision boundary for practical applications in context of random forests, given that a model was already selected. $\endgroup$
    – Scholar
    Commented Jan 22, 2019 at 16:09
  • $\begingroup$ I believe it is an exact duplicate. The OP has predicted probabilities and wants to convert them into classifications using thresholds. This is exactly what the proposed duplicate asks about. The difference is that we have multiple classes here and only two classes in the proposed duplicate, but this is minor. The answer in both cases is to assess the costs of misclassification and explicitly separate the probabilistic prediction from the decision. ... $\endgroup$ Commented Jan 22, 2019 at 16:17
  • $\begingroup$ ... (And that RFs may not output well-calibrated probabilistic predictions for fundamental reasons. I'll try to take a look at the Olson article you link in your answer, but to be honest, the abstract does not look convincing to me. In any case, the OP is treating the RF output as a bona fide probabilistic prediction, and this is at least a common conception about RFs. If this is incorrect systematically, then this is a separate - and interesting - question.) $\endgroup$ Commented Jan 22, 2019 at 16:21
  • $\begingroup$ @StephanKolassa You should look beyond the abstract then, they analyse probability estimates of random forests very rigorously. As for OPs question, I still don't think it's a duplicate. In the question you linked, the OP asks if changing the decision boundary makes for a better or worse model. Obviously, a model shouldn't evaluated on a single decision boundary and the final boundary is also subject to cost sensitivity, as you have explained nicely. $\endgroup$
    – Scholar
    Commented Jan 22, 2019 at 16:35
  • $\begingroup$ Anyways, once an optimal model was selected based on some scoring rule and the classifications costs have been defined for the specific use case, the model needs to be translate for practical use, i.e. probabilities (or scores) must be weighted and mapped to classes. IMO, OP is asking for a way to do exactly this final step in context of RF. $\endgroup$
    – Scholar
    Commented Jan 22, 2019 at 16:39

1 Answer 1

2
$\begingroup$

I don't think it's correct to talk about probabilities in context of random forests, as random forest classifiers do not attempt to produce accurate probabilities (see Olson et al. ). It's better to view the output as some score in the interval [0,1].

In random forest classification, each class $c_i, i \in {1, ..., k}$ gets assigned a score $s_i$ such that $\sum{s_i} = 1$. The model outputs the label of the class $c_i$ where $s_i = max({s_1, ..., s_k})$. So in order to adjust the thresholds, you can weight the scores $s_i$ by some weights $w_i$, such that you output the label of class $c_i$ with $s_i^* = max(s_1^*, ..., s_k^*) = max(s_1 \times w_1, ..., s_k \times w_k)$. (If you want the $s_i^*$ to add up to $1$, you need to normalize them.)


I'm not sure how or if answering questions such as "With maximal 1% probability we will be predicting B though it is actually A" is possible for > 2 classes, but for two classes, e.g. A and B, you can approach this by formulating a hypothesis test.

Given a sample X, in case of binary classification there are exactly two hypotheses in the universe. (1) X belongs to class A or (2) X belongs to class B. For simplicity, I will assume that the classes are balanced.

Sample X corresponds to a score $s_x \in [0, 1]$ and $s_x$ follows a different distribution depending whether the true class of X is A or B. Lets say that the mean score of samples in class A is bigger than the mean score of samples in class B, i.e. $\mu_A > \mu_B$.

Given your random forest model and a test set, you can calculate the empirical distribution of $s_x$ under A and $s_x$ under B.

Now, say you observed a sample X with score $s_x$. What is the probability that this class belongs to class A? To answer this, you can simply calculate the lower tail of the score-distribution under class A for $s_x$, i.e. the percentage of sample in class A with a score <= $s_x$. The resulting p-value corresponds to the probability that a sample X with score $s_x$ or lower truly belongs to the class A. You can do the same for class B by calculating the upper tail.

Similarly, for a given significance level $\alpha$, e.g. $\alpha = 0.01$, you can calculate a score $s_\alpha$ such that the chance that a sample X with score $s_x <= s_\alpha$ belongs to class A is less than $1\%$.

You see, since there are only two possible lables for a sample X, i.e. A or B, you can formulate a hypothesis test $H_0: X \in A, H_a: X \in B$ or vice verse.

Now, in case you have >2 classes, this is no longer possible, since you can only reject a given hypothesis, i.e. that a sample X belongs to A. In in the binary case, rejecting A corresponds to accepting B, since there are only two possible outcomes, but for e.g. 3 classes, rejecting A corresponds to accepting B and C!

Keep in mind that the procedure above only works for balanced classes. You can possibly extend this approach for the imbalanced case, but it will likely be more complicated and I just wanted to outline a possible general approach.

$\endgroup$
3
  • $\begingroup$ The question mainly is, how do I calculate the weights? Which algorithm do I use to find the weights so that I can say something like "With maximal 1% probability we will be predicting B though it is actually A" $\endgroup$
    – needRhelp
    Commented Jan 23, 2019 at 14:35
  • $\begingroup$ Given that I have a dataset with the three scores which the model predicted and the actual class $\endgroup$
    – needRhelp
    Commented Jan 23, 2019 at 14:38
  • $\begingroup$ @needRhelp see my updated answer. $\endgroup$
    – Scholar
    Commented Jan 23, 2019 at 15:19

Not the answer you're looking for? Browse other questions tagged or ask your own question.