2
$\begingroup$

I am new to the usage of a custom loss function for a model particularly for Xgboost and Lgbm. I have a binary classification problem which is highly imbalanced and I need to predict the probabilities for the minority class (1). For this the objective function I am using is objective = 'binary:logistic'. I did built an Xgboost model using the above ojective function and my evaluation metric being the average precision score. The score seems to be decent enough. But now I want to build a custom objective function for the model. So looking after many links and searching online, I used this as my custom objective function:

scale_pos_weight = 83

def obj_func(preds, y_train):
   weights = np.where(y_train == 1.0, scale_pos_weight, 1)   #as I use this parameter in my Xgbclassifier as well - to give weights to the minority class
   preds = 1.0 / (1.0 + np.exp(-preds))
   grad = preds - y_train       #gradient - 1st order derivative
   hess = preds * (1.0 - preds) #Hessian - 2nd order derivative
   return grad*weights, hess*weights

My Xgb classifier is defined as:

xgb = XGBClassifier(learning_rate =0.07,
                 n_estimators=1000,
                 max_depth=5,
                 gamma=2,
                 colsample_bytree=0.4,
                 objective= obj_func,
                 scale_pos_weight = 83,
                 seed=27)

model_xgb = xgb.fit(X_train, y_train)

After fitting the model I evaluate my model against the validation set using the average precision from sklearn.metrics. The evaluation was decent when I was using the default binary:logistic objective function from the library but after using the custom objective function the average precision has gotten worse (0.28) from (0.65 when using normal objective function). Is there something wrong that I am doing with my objective function or do I need to add something more?

$\endgroup$
1
  • $\begingroup$ If you are interested in predicting probabilities for the minority class then average precision is not a good performance metric - I would use the cross-entropy or Brier score. If you are interested in probabilities rather than hard classifications, you probably don't need to do anything about the class imbalance. $\endgroup$ Commented Jul 19, 2023 at 11:14

1 Answer 1

0
$\begingroup$

Maybe you could print your model_xgb.objective to check if it's the default one inferring from your input -- 'binary:logistics', if this is the case, and considering these two posts:

https://github.com/dmlc/xgboost/issues/5621

https://github.com/dmlc/xgboost/issues/3892

when you use a custom loss function with objective='binary:logistics', then you needn't do preds = 1.0 / (1.0 + np.exp(-preds)) in the udf loss function .

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.