I am training a random forest model to predict a certain outcome.
I split my data into an 80% training and 20% test set. In the training data, I used a grid-search to select optimal hyperparameters based on which hyper-parameters yielded the highest 5-fold cross AUC in this training set.
I then trained the model using these hyperparameters on the training set and determined the AUC on the test set.
I have a second external validation set which I also determined the AUC on.
Here are my results:
80% Training set AUC: 89 (95% CI: 85-92)
5-fold cross validation AUC in 80% training set: 64
20% Test set AUC: 80 (95% CI: 71-89)
External test set AUC: 70 (95% CI: 54-85)
My concern is that the 5 fold cross validation in the training set is so much lower than the other AUCs. I looked across several resources all of which suggested that the cross-validation AUC should only be used for selecting hyper-parameters and not for model evaluation.
Should I be concerned/distrustful of these results?