Hyper-parameter turning using cross validation in random forest----test set AUC higher than cross validated AUC

Question

I am training a random forest model to predict a certain outcome.

I split my data into an 80% training and 20% test set. In the training data, I used a grid-search to select optimal hyperparameters based on which hyper-parameters yielded the highest 5-fold cross AUC in this training set.

I then trained the model using these hyperparameters on the training set and determined the AUC on the test set.

I have a second external validation set which I also determined the AUC on.

Here are my results:

80% Training set AUC: 89 (95% CI: 85-92)

5-fold cross validation AUC in 80% training set: 64

20% Test set AUC: 80 (95% CI: 71-89)

External test set AUC: 70 (95% CI: 54-85)

My concern is that the 5 fold cross validation in the training set is so much lower than the other AUCs. I looked across several resources all of which suggested that the cross-validation AUC should only be used for selecting hyper-parameters and not for model evaluation.

Should I be concerned/distrustful of these results?

Answer 1

The difference is indeed significant, but it is hard to say without knowing the size of the dataset.
I can think of two possible reasons why you might have such a big difference:

• Your dataset is not particularly big, and therefore when running on a 5-Fold validation you are only using 4/5 of the 80% of your dataset (which is 64%) which might be insufficient for your RF to perform well

and/or

• Your dataset is not shuffled (idk how you are performing the train/test split nor the CV), and therefore issues very different performance based on the set it is training on

Finally, I would suggest you try using the Out Of Bag error of Random Forests instead of the 5 fold CV.
Basically this gives you an estimate of your TEST error (equivalent to a Cross Validation) by using the forecasts that your trees make on the data they were not trained on (because of the bootstrapping).
This is easier to do than a 5 fold CV, it does not reduce the size of the sample, and does not take any extra time. This will give you an extra estimate that you can compare to your test and then validation set results.

EDIT: I am also adding an extra information on how to cross-validate using the AUC, as it can be misleading: Appropriate way to get Cross Validated AUC