I trained a model with results as below. It is a stacking model with base learners of random forest and gradient boosting. The mega model is a GLM. The dataset is imbalanced in the target class as shown in the confusion matrix on the right top. The target class is a default status of a single loan (Positive: default; Negative: non-default).
The AUC ROC score is quite high but the f1 score is still only 0.53. My concern is both of recall and precision were approximately 0.5, meaning that the model can only distinguish half of the bad cases and only half of bad cases it diagnosed were truly bad.
If I adjust the probabilistic threshold, recall will increase but precision will also be sacrificed to some extent. Due to the highly imbalanced situation on the positive cases, a low precision may lead to a large proportion of False Negative among the predicted positive cases. e.g. under a lower precision, if a model tells 10 bad cases, there may be actually only 2 truly bad ones and more fake bad ones predicted which is not wanted in practice as well.
In predictive task on an imbalanced dataset, does f1 score matter and if so how can I further improve the score? (add new (composite) features, cost sensitive methods?)
Before asking this question, I have read the paper and get to know the skew of data does impact f1 score. But how to improve this case and reach a better predictive outcome for the practice.
