Why my model overfits despite selecting best hyperparameters value in each tuning step? I am fitting xgboost classification model to my data with highly inbalanced classes in response variable (99% vs 1%). I use cross-validation with k=5 to tune my hyperparameters:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, 
                                                    stratify=y, random_state=0)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

and end up with the following setup:
XGBClassifier(base_score=0.5, booster=None, colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=0.4,
              disable_default_eval_metric=1, gamma=0, gpu_id=-1,
              importance_type='gain', interaction_constraints=None,
              learning_rate=0.01, max_delta_step=0, max_depth=6,
              min_child_weight=4, missing=nan, monotone_constraints=None,
              n_estimators=5000, n_jobs=0, num_parallel_tree=1,
              objective='binary:logistic', random_state=42, reg_alpha=0,
              reg_lambda=10, scale_pos_weight=1, seed=42, subsample=1,
              tree_method=None, validate_parameters=False, verbosity=1)

Although above values were tuned I end up with the model that I would say overfit quite heavily:

From my understanding because of how gradient boosted trees works training data will always improve with more iterations. Test score seems to stabilise around 1500 iterations. But I don't like that huge difference between training and testing errors. Would you say I am overfitting here? If so, what am I doing wrong with my hyperparameters tuning if I choose best values for each parameter and end up with model that overfits anyway? Should I try to change for example gamma manually even despite gamma=0 was previously selected as the best?
As a side note - despite inbalanced classes I don't use scale_pos_weight because I mostly care about calibrated probabilites and not the exact predicted binary value.  
@Edit: I have added Train/Test split and Cross Validation details. The calibration plot (on the test set) looks like below:

 A: The key here is to understand what you are trying to achieve. Any statistical model will fit its training data better than testing data and if this is unacceptable for your case all you can really do is fit a null model.
In many cases, all we care about is the testing data performance. We don't use the training performance as an indication of model performance and just ignore training performance entirely.
"Overfitting" generally refers to the point where the model is so complex that the performance on testing data is compromised (you have not reached this point with your model yet).
This would be more complicated if the training and testing data weren't entirely independent (e.g. the data were clustered, or time series data, or similar) but in the case of independent observations.
Separately, you might wish to penalise the complexity of the model (in other words, you are prepared to accept slightly poorer testing performance if the model has fewer parameters in it). This will lead to less complex models which usually means a smaller difference between training and testing performance.
A: What is shown in the learning curves presented is not unprecedented within the context of boosting. It is widely observed that (especially in AdaBoost) the test error might decrease even after the training error is zero. See Schapire et al. (1998) Boosting the Margin: New Explanation for the Effectiveness of Voting Methods for details; the general idea is that maximizing the margin can improve the generalization error of a classifier even after the training error reaches zero. By "margin" we mean the distance between the sample point and the decision boundary learned by the classifier; we usually associate it with SVM (Support Vector Machines) but it is relevant for boosting too. 
Now, focusing again on the learning curve: over-fitting is split in two main situations: A. where both the training and test losses are decreasing, but the training loss is decreasing faster than the test loss and B. where the training loss is decreasing, but the test loss is increasing. The later (B) is clearly the problematic one for all classifiers. The former (A) is actually called optimism. Optimism is usually defined as the mean training error minus the mean validation error. Optimism of a model usually decreases with an increasing number of events per variable; van der Ploeg et al. (2014) Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints is an excellent and highly readable reference. Strictly speaking optimism is bad, but not the end of the world; it is more a problem with NN cause it suggests memorisation which in turn suggest issues with generalisation. That is because especially with a very large NN the capacity of it is sufficient for memorising the entire data set; Arpit et al. (2017) A Closer Look at Memorization in Deep Networks has more information on this. The same can happen with Gradient Boosting machines I suppose but I have not seen any references on the matter.
To recap, I think this model is mostly OK and does not over-fit massively, it just does not get enough "bang for its data buck" after some point! It appears to get rather optimistic after about 1000 iterations so it is worth exploring how to regularise it a bit more; for example, the subsample is set to 1 so it means we always used the whole training set when growing trees, maybe something smaller (0.80?) is more appropriate.
And a final note, the calibration plots looks good; no obvious S-shape and more or less looks monotonic. If not used already, using isotonic regression or a even a simple sigmoid on top of this classifier for some further probability calibration might help further both in terms of Brier score as well as with ranking measurements like AUC-ROC; it's no free lunch but it might help.
