75
$\begingroup$

I have a very imbalanced dataset. I'm trying to follow the tuning advice and use scale_pos_weight but not sure how should I tune it.

I can see that RegLossObj.GetGradient does:

if (info.labels[i] == 1.0f) w *= param_.scale_pos_weight

so a gradient of a positive sample would be more influential. However, according to the xgboost paper, the gradient statistic is always used locally = within the instances of a specific node in a specific tree:

  1. within the context of a node, to evaluate the loss reduction of a candidate split
  2. within the context of a leaf node, to optimize the weight given to that node

So there's no way of knowing in advance what would be a good scale_pos_weight - it is a very different number for a node that ends up with 1:100 ratio between positive and negative instances, and for a node with a 1:2 ratio.

Any hints?

Improve this question
$\endgroup$
3
  • $\begingroup$ Probably, you can tune the parameter in CV with 5 fold 5 repeats. But, you may need to write the code to do that. $\endgroup$
    – user2149631
    Commented Mar 7, 2017 at 8:21
  • $\begingroup$ Do you only want an answer for binary classification, or multiclass too? $\endgroup$
    – smci
    Commented Mar 13, 2020 at 2:53
  • $\begingroup$ Agreed, tuning the scale_pos_weight is the best approach. A simple grid search will quickly highlight values that do well. $\endgroup$
    – jasonb
    Commented May 15 at 20:51

4 Answers 4

Reset to default
72
$\begingroup$

Generally, scale_pos_weight is the ratio of number of negative class to the positive class.

Suppose, the dataset has 90 observations of negative class and 10 observations of positive class, then ideal value of scale_pos_weight should be 9.

See the doc: http://xgboost.readthedocs.io/en/latest/parameter.html

Improve this answer
$\endgroup$
11
  • 5
    $\begingroup$ How would that apply for a multiclass dataset? How about 28 classes? That's not clear to me $\endgroup$
    – Gabriel Ziegler
    Commented Feb 8, 2019 at 14:03
  • 2
    $\begingroup$ @Gabriel I believe then it would be better to go for class weights. You can use scale_pos_weight, by using one vs rest approach. For example, create dummies for 28 classes. Then you can use each one as a binary classification problem. That way you will be dealing with 28 different models. $\endgroup$
    – Harshit Mehta
    Commented Feb 8, 2019 at 16:02
  • 1
    $\begingroup$ I see, but when I use onevsrest doesn't the classifier also gives me a multilabel output, right? Not only one class out of the 28 $\endgroup$
    – Gabriel Ziegler
    Commented Feb 8, 2019 at 16:19
  • 1
    $\begingroup$ How ?. For Example: Classes are A,B,C. So you can have binary classifier for classifying (A/Not A ) , another one would be (B/Not B). You can do this for 'n' number of classes. Then among all the probabilities corresponding to each classifier, you have to find a way to assign classes. $\endgroup$
    – Harshit Mehta
    Commented Feb 8, 2019 at 16:48
  • 1
    $\begingroup$ I have just found this for those seeking for multiclass explanation datascience.stackexchange.com/questions/16342/… $\endgroup$
    – Gabriel Ziegler
    Commented Feb 8, 2019 at 18:47
20
$\begingroup$

All the documentation says that is should be:

scale_pos_weight = count(negative examples)/count(Positive examples)

In practice, that works pretty well, but if your dataset is extremely unbalanced I'd recommend using something more conservative like: 

scale_pos_weight = sqrt(count(negative examples)/count(Positive examples))

This is useful to limit the effect of a multiplication of positive examples by a very high weight.

Improve this answer
$\endgroup$
2
2
$\begingroup$

I understand your question and frustration, but I am not sure this is something that could be computed analytically, rather you'd have to determine a good setting empirically for your data, as you do for most hyper parameters, using cross validation as @user2149631 suggested. I've had some success using SelectFPR with Xgboost and the sklearn API to lower the FPR for XGBoost via feature selection instead, then further tuning the scale_pos_weight between 0 and 1.0. O.9 seems to work well but as with anything, YMMV depending on your data. You can also weight each data point individually when sending it to XGboost if you look through their docs. You have to use their API not the sklearn wrapper. That way you can weight one set of data points much higher than the other, and it will impact the boosting algorithm it uses.

Improve this answer
$\endgroup$
0
$\begingroup$

I also stumbled upon this dilemma and still looking for the best solution. However, I would suggest you using methods such as Grid Search (GridSearchCV in sklearn) for best parameter tuning for your classifier. However, if your dataset is highly imbalanced, its worthwhile to consider sampling methods (especially random oversampling and SMOTE oversampling methods) and model ensemble on data samples with different ratios of positive and negative class examples. Here is one nice and useful (almost comprehensive) tutorial about handling imbalanced datasets.

https://www.analyticsvidhya.com/blog/2017/03/imbalanced-classification-problem/

Improve this answer
$\endgroup$
1
  • $\begingroup$ I would recommend using RandomizedSearchCV in place of GridSearchCV, since the parameter space to search for boosted tree models is very large. Even better, perhaps, would be Bayesian grid search using Scikit-Optimize. (As I write this, skopt conflicts with Scikit-Learn v0.24, and would require downgrading the latter.) $\endgroup$
    – user3897315
    Commented Apr 14, 2021 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.