Cross Validation and test ROC AUC scores match but train score doesn't? I have a dataset of about ~49K entries and 31 columns. I ran a grid search with 3-fold CV for finding the hyperparameters of Random Forest (using sci-kit learn). I then used the best estimator to fit on the train set, and predict on the test set. The results of ROC AUC score are as follows:
CV: 0.705
Train: 0.836
Test: 0 .721
Can this be considered as overfitted? If so, what measures can I take to remedy this? So far, I have been spanning over n_estimators and max_depth. The model always seems to choose the maximum depth possible, and the difference between these scores increasing. I apply class weights to balance the dataset.
 A: You should plot the misclassification rate for the train and test, but my initial guess is that yes, you've overfitted the model. 
You could try tuning a XGBoost with different learning rates, and try out a different train proportions or/and minimum split sizes. Also try learning with as primitive weak learner as possible. 
A: It is difficult to say whether you are overfitting only looking at the ROCs for training and test. 
Even though it is true that the difference between the two is an estimate of the generalisation error, from a practical standpoint, you are overfitting when you reach that level of complexity that starts to deteriorate the out-of-sample performances of the model.
Two useful tools to diagnose overfitting are learning curves and validation curves (see here for a quick introduction).
As for the remedies:


*

*If possible, add more training data

*Perform feature selection 

*Since you are using a Random Forest, increasing the number of estimators will decrease the variance of your model, hence mitigating overfitting. See online or this question for the meaning of the hyperparameters, and their effect on the bias/variance tradeoff

*As suggested by @Nesvold, A good practice is to have a simpler, high bias, baseline model (e.g. a Logistic Regression) to compare your results with

