0
$\begingroup$

I am training a random forest model to predict a certain outcome.

I split my data into an 80% training and 20% test set. In the training data, I used a grid-search to select optimal hyperparameters based on which hyper-parameters yielded the highest 5-fold cross AUC in this training set.

I then trained the model using these hyperparameters on the training set and determined the AUC on the test set.

I have a second external validation set which I also determined the AUC on.

Here are my results:

80% Training set AUC: 89 (95% CI: 85-92)

5-fold cross validation AUC in 80% training set: 64

20% Test set AUC: 80 (95% CI: 71-89)

External test set AUC: 70 (95% CI: 54-85)

My concern is that the 5 fold cross validation in the training set is so much lower than the other AUCs. I looked across several resources all of which suggested that the cross-validation AUC should only be used for selecting hyper-parameters and not for model evaluation.

Should I be concerned/distrustful of these results?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

The difference is indeed significant, but it is hard to say without knowing the size of the dataset.
I can think of two possible reasons why you might have such a big difference:

  • Your dataset is not particularly big, and therefore when running on a 5-Fold validation you are only using 4/5 of the 80% of your dataset (which is 64%) which might be insufficient for your RF to perform well

and/or

  • Your dataset is not shuffled (idk how you are performing the train/test split nor the CV), and therefore issues very different performance based on the set it is training on

Finally, I would suggest you try using the Out Of Bag error of Random Forests instead of the 5 fold CV.
Basically this gives you an estimate of your TEST error (equivalent to a Cross Validation) by using the forecasts that your trees make on the data they were not trained on (because of the bootstrapping).
This is easier to do than a 5 fold CV, it does not reduce the size of the sample, and does not take any extra time. This will give you an extra estimate that you can compare to your test and then validation set results.

EDIT: I am also adding an extra information on how to cross-validate using the AUC, as it can be misleading: Appropriate way to get Cross Validated AUC

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you so much and sorry for the late reply. I have 1020 datapoints in my training set. only 10% of the datapoints contains the variable that I am trying to predict for (this is a patient dataset and I am predicting cancer patient survival--only 10% of patients died, creating an unbalanced dataset) $\endgroup$
    – newbie101
    Dec 11, 2021 at 22:42
  • $\begingroup$ Additionally, my oob error is ~40%....which is really really high.....any tips on how to improve this in a random forest survival model? I'm using the randomForestSRC package. $\endgroup$
    – newbie101
    Dec 11, 2021 at 22:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.