20
$\begingroup$

I have a very imbalanced dataset. I'm trying to follow the tuning advice and use scale_pos_weight but not sure how should I tune it.

I can see that RegLossObj.GetGradient does:

if (info.labels[i] == 1.0f) w *= param_.scale_pos_weight

so a gradient of a positive sample would be more influential. However, according to the xgboost paper, the gradient statistic is always used locally = within the instances of a specific node in a specific tree:

  1. within the context of a node, to evaluate the loss reduction of a candidate split
  2. within the context of a leaf node, to optimize the weight given to that node

So there's no way of knowing in advance what would be a good scale_pos_weight - it is a very different number for a node that ends up with 1:100 ratio between positive and negative instances, and for a node with a 1:2 ratio.

Any hints?

$\endgroup$
  • $\begingroup$ Probably, you can tune the parameter in CV with 5 fold 5 repeats. But, you may need to write the code to do that. $\endgroup$ – user2149631 Mar 7 '17 at 8:21
14
$\begingroup$

Generally, the Scale_pos_weight is the ratio of number of negative class to the positive class. Suppose, the dataset has 90 observations of negative class and 10 observations of positive class, then ideal value of scale_pos_Weight should be 9. You can check the following link. http://xgboost.readthedocs.io/en/latest/parameter.html

$\endgroup$
  • $\begingroup$ How would that apply for a multiclass dataset? How about 28 classes? That's not clear to me $\endgroup$ – Gabriel Ziegler Feb 8 at 14:03
  • $\begingroup$ @Gabriel I believe then it would be better to go for class weights. You can use scale_pos_weight, by using one vs rest approach. For example, create dummies for 28 classes. Then you can use each one as a binary classification problem. That way you will be dealing with 28 different models. $\endgroup$ – Harshit Mehta Feb 8 at 16:02
  • $\begingroup$ I see, but when I use onevsrest doesn't the classifier also gives me a multilabel output, right? Not only one class out of the 28 $\endgroup$ – Gabriel Ziegler Feb 8 at 16:19
  • $\begingroup$ How ?. For Example: Classes are A,B,C. So you can have binary classifier for classifying (A/Not A ) , another one would be (B/Not B). You can do this for 'n' number of classes. Then among all the probabilities corresponding to each classifier, you have to find a way to assign classes. $\endgroup$ – Harshit Mehta Feb 8 at 16:48
  • $\begingroup$ I'm used to using onevsrest for multilabel, but I'll give it a try! Thanks! $\endgroup$ – Gabriel Ziegler Feb 8 at 18:39
1
$\begingroup$

I understand your question and frustration, but I am not sure this is something that could be computed analytically, rather you'd have to determine a good setting empirically for your data, as you do for most hyper parameters, using cross validation as @user2149631 suggested. I've had some success using SelectFPR with Xgboost and the sklearn API to lower the FPR for XGBoost via feature selection instead, then further tuning the scale_pos_weight between 0 and 1.0. O.9 seems to work well but as with anything, YMMV depending on your data. You can also weight each data point individually when sending it to XGboost if you look through their docs. You have to use their API not the sklearn wrapper. That way you can weight one set of data points much higher than the other, and it will impact the boosting algorithm it uses.

$\endgroup$
0
$\begingroup$

I also stumbled upon this dilemma and still looking for the best solution. However, I would suggest you using methods such as Grid Search (GridSearchCV in sklearn) for best parameter tuning for your classifier. However, if your dataset is highly imbalanced, its worthwhile to consider sampling methods (especially random oversampling and SMOTE oversampling methods) and model ensemble on data samples with different ratios of positive and negative class examples. Here is one nice and useful (almost comprehensive) tutorial about handling imbalanced datasets.

https://www.analyticsvidhya.com/blog/2017/03/imbalanced-classification-problem/

$\endgroup$
0
$\begingroup$

All the documentation says that is should be:

scale_pos_weight = count(negative examples)/count(Positive examples)

In practice, that works pretty well, but if your dataset is extremely unbalanced I'd recommend using something more conservative like:

scale_pos_weight = sqrt(count(negative examples)/count(Positive examples)) 

This is useful to limit the effect of a multiplication of positive examples by a very high weight.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.