Model has high ROC Value in Cross Validation but performs poorly on unseen data

Question

Im running a Random Forest to classify a binary outcome in R. I use k fold Cross Validation to determine the best model features (mtry) and choose the best model based off the highest ROC value. My ROC values are high, see below:

 mtry  ROC        Sens       Spec     

1 0.7874624 0.4661538 0.9264744

2 0.8798629 0.6280128 0.9872436

3 0.9658186 0.7065385 1.0000000

4 0.9788579 0.8607051 1.0000000

5 0.9837602 0.8835256 1.0000000

6 0.9851584 0.8886538 1.0000000

When i run my model with 6 mtry as suggested by the k fold Cross validation and test this model on the test set i get very poor performance, see below confusion mmatrix:

    actual
predictions  No Yes
        No  139  19
        Yes  27   2         

I thought k fold Cross Validation is a method that can be used to reduce the overfitting issue and guide you in choosing the correct model. However when i pick the best model suggested by k fold Cross Validation the model is extremly poor at predicting unseen data. I have two questions:

1) Is the model not predicting unseen data because the model is overfit to the train data

2) What is the point in k fold Cross Validation if overfitting is still an issue - i.e. why not just use the traditional method of hold one out with a train/test split?

Addition: Here is my code whcih shows the variables i am using.

modelrf3<-train(Turtle_factor~Blue+Green+Mesh+Twine+NE+SW,data=overrf1,method='rf',metric="ROC",trControl=ControlParameters,tuneGrid=parameterGrid)

enter code here

variables:

Blue, Green are categorical - colour

Mesh and Twine are continuos and measure the size of a fishing net in mm

NE and SW are categorical and represent two different seasons (North East and South West Monsoon)

Answer 1

Is the model not predicting unseen data because the model is overfit to the train data

Possibly. But it may also just be a mix of

• the relative frequency of your classes is about 8 : 1 = 11 % of class Yes in the test data you show

• accidentally bad single split into train and test: class Yes overrepresented in the test set and underrepresented in the train/optimization sets
  (you don't say that you stratified the splitting)

• There is no indication how the working point (threshold for assigning class labels) was chosen. With so unequal relative frequencies and comparably low numbers of cases, you need to think about this.
• Also, the area under the ROC may not be the value you want to optimize - it may be better to optimize a figure of merit that is tailored to your application and that includes fixing the working point. If you want to optimze hyperparameters like mtry, go for a proper scoring rule.

What is the point in k fold Cross Validation if overfitting is still an issue - i.e. why not just use the traditional method of hold one out with a train/test split?

• The advantages of cross validation over a single random split are that you can check stability and that it does make more efficient use of the (few) cases you have: precision of cross validation estimates is better as more cases are tested.

• cross-validation is not a magic wand against overfitting. If the overfitting is caused by clusters in the data (i.e. not all rows are statistically independent), then random splits will not be able to guard against such overfitting. Neither via cross validation nor via random splits into hold out sets.
  A good test set (e.g. produced by a separate experiment) will be able to spot this issue.

• Cross validation reduces overfitting compared to training-set performance estimates (i.e. pure goodness-of-fit), and it is somewhat better than single split hold out because of the reduced variance (which translates to overfitting bias after systematic optimization).

---

• BTW, neither is the random forest a magic wand: bagging helps only if the issue is instability of the model - if that isn't the issue, then bagging doesn't improve predictions. So do a sanity check: does the random forest predict mostly probabilities very close to 0 and 1 or do you get intermediate values as well?