Does it make sense to use Focal loss for a tree based classifier like XGBoost?

Question

I am working on a classification problem with an imbalanced data (0.2% of positive class). I've seen people using Focal Loss in deep learning in order to help the model focus on hard examples more. I'd like to know if this objective function could be used in a tree based model like Xgboost or LightGBM, any literature on the subject would be welcome as well.

Answer 1

Sampling and tree-based classifiers go hand-in-hand, so my first impression is yes. There are a variety of sampling techniques (already available) that you could consider in tandem.

The classic imbalanced sampling technique is SMOTE (see ref below), which oversamples from the minority class to synthetically increase its prevalence. Boosting algorithms (like adaboost)also will oversample the cases it got wrong, in order to fix issues with predictions. Focal is similar in that it will down-weight the "easy" predictors (in the loss function), so it makes sense to use it. The tricky part is that boosting algorithms are essentially prone to overfitting since their sampling is gradient-based to reduce error, so one must be always careful with how to introduce sampling schemes and loss functions. That's the only caveat with them. Below I've included all 3 references.

SMOTE: Chawla, Nitesh V., Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. "SMOTE: synthetic minority over-sampling technique." Journal of artificial intelligence research 16 (2002): 321-357.

Adaboost: Rätsch, Gunnar, Takashi Onoda, and K-R. Müller. "Soft margins for AdaBoost." Machine learning 42, no. 3 (2001): 287-320.

Focal: Lin, T. Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision (pp. 2980-2988).

Hope this helps