I have a very imbalanced dataset. I'm trying to follow the tuning advice and use
scale_pos_weight but not sure how should I tune it.
I can see that
if (info.labels[i] == 1.0f) w *= param_.scale_pos_weight
so a gradient of a positive sample would be more influential. However, according to the xgboost paper, the gradient statistic is always used locally = within the instances of a specific node in a specific tree:
- within the context of a node, to evaluate the loss reduction of a candidate split
- within the context of a leaf node, to optimize the weight given to that node
So there's no way of knowing in advance what would be a good
scale_pos_weight - it is a very different number for a node that ends up with 1:100 ratio between positive and negative instances, and for a node with a 1:2 ratio.