ROC curve image on train and test data

I try to classify data from a dataset of 35K data point and 12 features

Firstly i have divided the data into train and test data for cross validation After cross validation i have built a XGBoost model using below parameters

n_estimators = 100


scale_pos_weight = 0.2 as data is imbalanced(85%positive class)

But model is overfitting the train data. what can be done to avoid overfitting?


XGBoost (and other gradient boosting machine routines too) has a number of parameters that can be tuned to avoid over-fitting. I will mention some of the most obvious ones. For example we can change:

  • the ratio of features used (i.e. columns used); colsample_bytree. Lower ratios avoid over-fitting.
  • the ratio of the training instances used (i.e. rows used); subsample. Lower ratios avoid over-fitting.
  • the maximum depth of a tree; max_depth. Lower values avoid over-fitting.
  • the minimum loss reduction required to make a further split; gamma. Larger values avoid over-fitting.
  • the learning rate of our GBM (i.e. how much we update our prediction with each successive tree); eta. Lower values avoid over-fitting.
  • the minimum sum of instance weight needed in a leaf, in certain applications this relates directly to the minimum number of instances needed in a node; min_child_weight. Larger values avoid over-fitting.

This list is not exhaustive and I will strongly urge looking into XGBoost docs for information regarding other parameters. Please note that trying to avoid over-fitting might lead to under-fitting, where we regularise too much and fail to learn relevant information. On that matter, one might want to consider using a separate validation set or simply cross-validation (through xgboost.cv() for example) to monitor the progress of the GBM as more iterations are performed (i.e. base learners are added). That way potentially over-fitting problems can be caught early on. This relates close to the use of early-stopping as a form a regularisation; XGBoost offers an argument early_stopping_rounds that is relevant in this case.

Finally, I would also note that the class imbalance reported (85-15) is not really severe. Using the default value scale_pos_weight of 1 is probably adequate.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.