I trained a model with results as below. It is a stacking model with base learners of random forest and gradient boosting. The mega model is a GLM. The dataset is imbalanced in the target class as shown in the confusion matrix on the right top. The target class is a default status of a single loan (Positive: default; Negative: non-default).

The AUC ROC score is quite high but the f1 score is still only 0.53. My concern is both of recall and precision were approximately 0.5, meaning that the model can only distinguish half of the bad cases and only half of bad cases it diagnosed were truly bad.

If I adjust the probabilistic threshold, recall will increase but precision will also be sacrificed to some extent. Due to the highly imbalanced situation on the positive cases, a low precision may lead to a large proportion of False Negative among the predicted positive cases. e.g. under a lower precision, if a model tells 10 bad cases, there may be actually only 2 truly bad ones and more fake bad ones predicted which is not wanted in practice as well.

In predictive task on an imbalanced dataset, does f1 score matter and if so how can I further improve the score? (add new (composite) features, cost sensitive methods?)

enter image description here

Before asking this question, I have read the paper and get to know the skew of data does impact f1 score. But how to improve this case and reach a better predictive outcome for the practice.

enter image description here

  • 1
    $\begingroup$ To me "better predictive outcome for the practice" equates to "better performance for the real-life use". ie. assign expected costs (or gains depending which ones you want to be considered having negative values) for FP, FN, TP and TN and show that. All other are standard fluffy stuff that we show when we don't care a lot about practice or we don't bother/care to find specific costs but really we just want to "prove" one classifier better than another in a particular scenario. (I specifically say "we" because I also do it when I first visit a problem.) $\endgroup$ – usεr11852 says Reinstate Monic Apr 20 '17 at 23:41
  • $\begingroup$ Thanks @usεr11852 and I also agree with the cost sensitive methods which is more approaching to the reality. $\endgroup$ – LUSAQX Apr 20 '17 at 23:51
  • 3
    $\begingroup$ Cross-posted: datascience.stackexchange.com/q/18495/8560, stats.stackexchange.com/q/274870/2921. Please do not post the same question on multiple sites. Each community should have an honest shot at answering without anybody's time being wasted. $\endgroup$ – D.W. Jun 28 '17 at 4:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.