I am working on imbalanced dataset. I am usng three algorithms: RF, SVM and J48. Generally an instance is classified as positive if its classification score is greater than 0.5. However, since I am working on imbalanced data, I perform a small experiment. I compute F-measure of all the classifiers at different decision thresholds form 0.1 to 0.9. I found that RF is most sensitive to the decision threshold. Does any one have any idea why this is happening?
You need to consider that the different methods have different degree of inductive bias. Some methods are rather variabel (trees, RF,...) and tend to be able to fit complex decision boundaries. Other methods are less variable and tend have more bias (see also vatiance-bias decomposition for the 0-1 loss which you can run from the python package mlxtend https://github.com/rasbt/mlxtend/blob/master/mlxtend/evaluate/bias_variance_decomp.py#L19).
Please also consider that many classifiers are not calibrated. I.e. a score of 0.6 does not have to be more telling than a score of 0.9 with regards to telling you that the sample belongs to class 1.
All this makes RF easily toggle the classification label when you change the decision threshold. You will be able to see this in the precision recall and ROC curves too.
The problem is emphasized for very rare classes if you compute the recall or precision or fscore for this class because changing one classification outcome has big impact on those values (which you can verify by propagation of uncertainty for e.g. the recall or precision formula).