# how to avoid overfitting in XGBoost model

I try to classify data from a dataset of 35K data point and 12 features

Firstly i have divided the data into train and test data for cross validation After cross validation i have built a XGBoost model using below parameters

n_estimators = 100

max_depth=4

scale_pos_weight = 0.2 as data is imbalanced(85%positive class)

But model is overfitting the train data. what can be done to avoid overfitting?

XGBoost (and other gradient boosting machine routines too) has a number of parameters that can be tuned to avoid over-fitting. I will mention some of the most obvious ones. For example we can change:

• the ratio of features used (i.e. columns used); colsample_bytree. Lower ratios avoid over-fitting.
• the ratio of the training instances used (i.e. rows used); subsample. Lower ratios avoid over-fitting.
• the maximum depth of a tree; max_depth. Lower values avoid over-fitting.
• the minimum loss reduction required to make a further split; gamma. Larger values avoid over-fitting.
• the learning rate of our GBM (i.e. how much we update our prediction with each successive tree); eta. Lower values avoid over-fitting.
• the minimum sum of instance weight needed in a leaf, in certain applications this relates directly to the minimum number of instances needed in a node; min_child_weight. Larger values avoid over-fitting.

This list is not exhaustive and I will strongly urge looking into XGBoost docs for information regarding other parameters. Please note that trying to avoid over-fitting might lead to under-fitting, where we regularise too much and fail to learn relevant information. On that matter, one might want to consider using a separate validation set or simply cross-validation (through xgboost.cv() for example) to monitor the progress of the GBM as more iterations are performed (i.e. base learners are added). That way potentially over-fitting problems can be caught early on. This relates close to the use of early-stopping as a form a regularisation; XGBoost offers an argument early_stopping_rounds that is relevant in this case.

Finally, I would also note that the class imbalance reported (85-15) is not really severe. Using the default value scale_pos_weight of 1 is probably adequate.

• +1. Another thing that might help is to be sure that the class imbalance is equivalent between the training set and testing set by using stratified sampling with the outcome when making the split. Even moderate discrepancies due to an unlucky split without stratification can yield results similar to what you have in your plot. – dmartin Sep 12 at 19:53
• @dmartin: Thank you for you upvote but apologies as I somewhat disagree with the point you make. Unless we are looking at a severely imbalanced problem a performance degradation in terms of AUC-ROC from 90% down to 68% is extremely unlikely to be due to "moderate discrepancies" in the train-test (T-T) split. Using stratified sampling is always a reasonable idea. For the case presented though, 35K / 85-15 class imbalance, the chance of a say at 80-20 T-T split more disproportionate than an 87.5-12.5 class split is next to none. (cont.) – usεr11852 Sep 12 at 21:43
• Taking in account also that AUC-ROC is reasonably robust to class imbalances, I think the train-test split is very unlikely to be the culprit. – usεr11852 Sep 12 at 21:44
• It certainly is not likely, but something to check. The reason I commented was that I recently saw a case where a bug in someone's code caused them to erroneously have an 80-20 imbalance in train and a 90-10 in test. The AUC plot was very similar to what was posted in the question (AUC of .85 going down to .7). After spending quite some time tuning the xgboost parameters to reduce complexity with no avail, I had them check the imbalance and they found this issue. Using stratified sampling fixed it entirely. – dmartin Sep 13 at 22:42
• @dmartin: Thank you for the clarification, I stand corrected it seems. Must have been a pretty unlucky run. Even with 1000 points, when starting with 80-20 class imbalance, the chance of getting a 90-10 class split given a 80-20 T-T split, is less than 0.01%. – usεr11852 Sep 14 at 0:06