# how to avoid overfitting in XGBoost model

I try to classify data from a dataset of 35K data point and 12 features

Firstly i have divided the data into train and test data for cross validation After cross validation i have built a XGBoost model using below parameters

n_estimators = 100

max_depth=4

scale_pos_weight = 0.2 as data is imbalanced(85%positive class)

But model is overfitting the train data. what can be done to avoid overfitting?

• the ratio of features used (i.e. columns used); colsample_bytree. Lower ratios avoid over-fitting.
• the ratio of the training instances used (i.e. rows used); subsample. Lower ratios avoid over-fitting.
• the maximum depth of a tree; max_depth. Lower values avoid over-fitting.
• the minimum loss reduction required to make a further split; gamma. Larger values avoid over-fitting.
• the learning rate of our GBM (i.e. how much we update our prediction with each successive tree); eta. Lower values avoid over-fitting.
• the minimum sum of instance weight needed in a leaf, in certain applications this relates directly to the minimum number of instances needed in a node; min_child_weight. Larger values avoid over-fitting.
This list is not exhaustive and I will strongly urge looking into XGBoost docs for information regarding other parameters. Please note that trying to avoid over-fitting might lead to under-fitting, where we regularise too much and fail to learn relevant information. On that matter, one might want to consider using a separate validation set or simply cross-validation (through xgboost.cv() for example) to monitor the progress of the GBM as more iterations are performed (i.e. base learners are added). That way potentially over-fitting problems can be caught early on. This relates close to the use of early-stopping as a form a regularisation; XGBoost offers an argument early_stopping_rounds that is relevant in this case.
Finally, I would also note that the class imbalance reported (85-15) is not really severe. Using the default value scale_pos_weight of 1 is probably adequate.