I want to understand whether giving weights to instances across a dataset in XGBoost using the below method makes sense. I switched to this method after trying out a few approaches that didn't fare well (ex: giving weights to class instances depending on event rate- similar to
scale_pos_weight, etc.). I am using the following method only for binary classification
- run an xgboost model without any instance weights
- get the probability of the event for each instance.
Calculate the distance between the probability and
0.5. The idea is to give more weight to instances that have probability closer to the threshold,
0.5, than the ones farther away. This is how I am calculating weights.
if P(event) <= 0.5: weight = P(event)*2
else: weight = (1 - P(event))*2
run next iterations using results from the previous run to generate weights
The reason behind using the weights this way was to try to keep the model away from instances with very high/very low probability. My interpretation is that the model is fairly confident about those instances and hence should focus elsewhere.
I have run a few experiments (on the adult dataset) and have found that adding weights in this manner does improve the model fit by a few points.
I want to understand how generalizable this approach is. And what potential issues I could be facing using the above method. I am not very familiar with XGBoost internals and need help understanding the implications of the above approach.