XGBoost (and other gradient boosting machine routines too) has a number of parameters that can be tuned to avoid over-fitting. I will mention some of the most obvious ones. For example we can change:
- the ratio of features used (i.e. columns used);
colsample_bytree. Lower ratios avoid over-fitting.
- the ratio of the training instances used (i.e. rows used);
subsample. Lower ratios avoid over-fitting.
- the maximum depth of a tree;
max_depth. Lower values avoid over-fitting.
- the minimum loss reduction required to make a further split;
gamma. Larger values avoid over-fitting.
- the learning rate of our GBM (i.e. how much we update our prediction with each successive tree);
eta. Lower values avoid over-fitting.
- the minimum sum of instance weight needed in a leaf, in certain applications this relates directly to the minimum number of instances needed in a node;
min_child_weight. Larger values avoid over-fitting.
This list is not exhaustive and I will strongly urge looking into XGBoost docs for information regarding other parameters. Please note that trying to avoid over-fitting might lead to under-fitting, where we regularise too much and fail to learn relevant information. On that matter, one might want to consider using a separate validation set or simply cross-validation (through
xgboost.cv() for example) to monitor the progress of the GBM as more iterations are performed (i.e. base learners are added). That way potentially over-fitting problems can be caught early on. This relates close to the use of early-stopping as a form a regularisation; XGBoost offers an argument
early_stopping_rounds that is relevant in this case.
Finally, I would also note that the class imbalance reported (85-15) is not really severe. Using the default value
scale_pos_weight of 1 is probably adequate.