Im running a Random Forest to classify a binary outcome in R. I use k fold Cross Validation to determine the best model features (mtry) and choose the best model based off the highest ROC value. My ROC values are high, see below:

 mtry  ROC        Sens       Spec     

1 0.7874624 0.4661538 0.9264744

2 0.8798629 0.6280128 0.9872436

3 0.9658186 0.7065385 1.0000000

4 0.9788579 0.8607051 1.0000000

5 0.9837602 0.8835256 1.0000000

6 0.9851584 0.8886538 1.0000000

When i run my model with 6 mtry as suggested by the k fold Cross validation and test this model on the test set i get very poor performance, see below confusion mmatrix:

predictions  No Yes
        No  139  19
        Yes  27   2         

I thought k fold Cross Validation is a method that can be used to reduce the overfitting issue and guide you in choosing the correct model. However when i pick the best model suggested by k fold Cross Validation the model is extremly poor at predicting unseen data. I have two questions:

1) Is the model not predicting unseen data because the model is overfit to the train data

2) What is the point in k fold Cross Validation if overfitting is still an issue - i.e. why not just use the traditional method of hold one out with a train/test split?

Addition: Here is my code whcih shows the variables i am using.


enter code here


Blue, Green are categorical - colour

Mesh and Twine are continuos and measure the size of a fishing net in mm

NE and SW are categorical and represent two different seasons (North East and South West Monsoon)

  • $\begingroup$ It does sound like overfitting. Have you tried gradually increasing the number of trees in the forest? $\endgroup$
    – Digio
    Commented Jul 11, 2017 at 13:35
  • $\begingroup$ Yes i tried this but nothing happens with the ROC. Correct me if im wrong but i thought Cross Validation reduces overfitting? $\endgroup$
    – Martin
    Commented Jul 11, 2017 at 13:36
  • $\begingroup$ RF's bootstrap aggregation reduces variance, yes, but that doesn't mean it is a magic wand. Have you tried something like logistic regression first? If your data is linearly separable, then it will be prone to overfitting if you use a nonlinear method such as RF. $\endgroup$
    – Digio
    Commented Jul 11, 2017 at 13:38
  • $\begingroup$ Thanks Digio i have tried logistic regression and my variables are not linear. $\endgroup$
    – Martin
    Commented Jul 11, 2017 at 13:40
  • 1
    $\begingroup$ With ROC you presumably mean the area under the ROC (AuROC)? $\endgroup$
    – cbeleites
    Commented Jul 13, 2017 at 7:25

1 Answer 1


Is the model not predicting unseen data because the model is overfit to the train data

Possibly. But it may also just be a mix of

  • the relative frequency of your classes is about 8 : 1 = 11 % of class Yes in the test data you show

  • accidentally bad single split into train and test: class Yes overrepresented in the test set and underrepresented in the train/optimization sets
    (you don't say that you stratified the splitting)

  • There is no indication how the working point (threshold for assigning class labels) was chosen. With so unequal relative frequencies and comparably low numbers of cases, you need to think about this.
  • Also, the area under the ROC may not be the value you want to optimize - it may be better to optimize a figure of merit that is tailored to your application and that includes fixing the working point. If you want to optimze hyperparameters like mtry, go for a proper scoring rule.

What is the point in k fold Cross Validation if overfitting is still an issue - i.e. why not just use the traditional method of hold one out with a train/test split?

  • The advantages of cross validation over a single random split are that you can check stability and that it does make more efficient use of the (few) cases you have: precision of cross validation estimates is better as more cases are tested.

  • cross-validation is not a magic wand against overfitting. If the overfitting is caused by clusters in the data (i.e. not all rows are statistically independent), then random splits will not be able to guard against such overfitting. Neither via cross validation nor via random splits into hold out sets.
    A good test set (e.g. produced by a separate experiment) will be able to spot this issue.

  • Cross validation reduces overfitting compared to training-set performance estimates (i.e. pure goodness-of-fit), and it is somewhat better than single split hold out because of the reduced variance (which translates to overfitting bias after systematic optimization).

  • BTW, neither is the random forest a magic wand: bagging helps only if the issue is instability of the model - if that isn't the issue, then bagging doesn't improve predictions. So do a sanity check: does the random forest predict mostly probabilities very close to 0 and 1 or do you get intermediate values as well?
  • $\begingroup$ Thanks cbeleites. FYI i did use stratified sampling during the train/split test however even so there may still be an different values in test set that are not represented in train. Due to the class imbalance i decided to go for ROC metric because it gives a true false positive/false negative score as opposed to accuracy that favours the majority class. Also im using ROSE function in R to randomly oversample the minority class to see if the predictive model is better. Im not hopeful as i think i may have a unpredictable event $\endgroup$
    – Martin
    Commented Jul 14, 2017 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.