1
$\begingroup$

I have a dataset of about ~49K entries and 31 columns. I ran a grid search with 3-fold CV for finding the hyperparameters of Random Forest (using sci-kit learn). I then used the best estimator to fit on the train set, and predict on the test set. The results of ROC AUC score are as follows:

CV: 0.705 Train: 0.836 Test: 0 .721

Can this be considered as overfitted? If so, what measures can I take to remedy this? So far, I have been spanning over n_estimators and max_depth. The model always seems to choose the maximum depth possible, and the difference between these scores increasing. I apply class weights to balance the dataset.

Improve this question
$\endgroup$
3
  • $\begingroup$ Once you've found the hyperparameters, when you say "test set", you mean some hold out data that you haven't used for CV? Thanks. $\endgroup$
    – lrnzcig
    Commented Apr 13, 2017 at 10:37
  • $\begingroup$ Yes, it's a holdout set that is completely unaware of the training process. $\endgroup$
    – infinite-rotations
    Commented Apr 13, 2017 at 15:51
  • $\begingroup$ Then your CV auc is actually smaller than the holdout auc right? Anyway from your description I would try to investigate what's happening with the max_depth. Even if you put something huge, your CV chooses it? $\endgroup$
    – lrnzcig
    Commented Apr 13, 2017 at 18:06

2 Answers 2

Reset to default
1
$\begingroup$

You should plot the misclassification rate for the train and test, but my initial guess is that yes, you've overfitted the model.

You could try tuning a XGBoost with different learning rates, and try out a different train proportions or/and minimum split sizes. Also try learning with as primitive weak learner as possible.

Improve this answer
$\endgroup$
1
$\begingroup$

It is difficult to say whether you are overfitting only looking at the ROCs for training and test.

Even though it is true that the difference between the two is an estimate of the generalisation error, from a practical standpoint, you are overfitting when you reach that level of complexity that starts to deteriorate the out-of-sample performances of the model.

Two useful tools to diagnose overfitting are learning curves and validation curves (see here for a quick introduction).

As for the remedies:

  • If possible, add more training data
  • Perform feature selection
  • Since you are using a Random Forest, increasing the number of estimators will decrease the variance of your model, hence mitigating overfitting. See online or this question for the meaning of the hyperparameters, and their effect on the bias/variance tradeoff
  • As suggested by @Nesvold, A good practice is to have a simpler, high bias, baseline model (e.g. a Logistic Regression) to compare your results with
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.