2
$\begingroup$

I am working on imbalanced dataset. I am usng three algorithms: RF, SVM and J48. Generally an instance is classified as positive if its classification score is greater than 0.5. However, since I am working on imbalanced data, I perform a small experiment. I compute F-measure of all the classifiers at different decision thresholds form 0.1 to 0.9. I found that RF is most sensitive to the decision threshold. Does any one have any idea why this is happening?

$\endgroup$
3
  • 2
    $\begingroup$ What do you mean by "sensitive to classification score"? Sensitivity usually means the true-positive rate, but you use the F1-score. $\endgroup$
    – dimpol
    Oct 31, 2016 at 10:32
  • $\begingroup$ I mean Random forest is showing lots of variation in F-score by just changing classification threshold form 0.5 to 0.3. $\endgroup$
    – Sangeeta
    Oct 31, 2016 at 16:15
  • $\begingroup$ We get a lot of questions on here about the many misconceptions concerning class imbalance, leading to this Meta post. I believe this question to be distinct. $\endgroup$
    – Dave
    Aug 10, 2022 at 20:13

1 Answer 1

0
$\begingroup$

You need to consider that the different methods have different degree of inductive bias. Some methods are rather variabel (trees, RF,...) and tend to be able to fit complex decision boundaries. Other methods are less variable and tend have more bias (see also vatiance-bias decomposition for the 0-1 loss which you can run from the python package mlxtend https://github.com/rasbt/mlxtend/blob/master/mlxtend/evaluate/bias_variance_decomp.py#L19).

Please also consider that many classifiers are not calibrated. I.e. a score of 0.6 does not have to be more telling than a score of 0.9 with regards to telling you that the sample belongs to class 1.

All this makes RF easily toggle the classification label when you change the decision threshold. You will be able to see this in the precision recall and ROC curves too.

The problem is emphasized for very rare classes if you compute the recall or precision or fscore for this class because changing one classification outcome has big impact on those values (which you can verify by propagation of uncertainty for e.g. the recall or precision formula).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.