I'm building a binary classifier, where each record is a task, and the response variable is whether it was completed on time.

I'm using random forest My data set spans from 2000-2015

My hold out test set has far worse performance than my validation set

#Get last year of available data as test data
DataTest <- Data %>% filter(DATE_YEAR == 2015)
#The rest of the data is for training and validation
DataTrainAndVs <- Data %>% filter(DATE_YEAR != 2015)

So I'm keeping the year 2015 as a hold out test set

#Here I take 85% of the data from 2000 to 2014 to train with, 
#and 15% of the 2000-2014 data will be for validation and tuning
PRCNT_TR <- 0.85
DataTrainIndex <- sample(1:nrow(DataTrainAndVs), 
                         round(PRCNT_TR * nrow(DataTrainAndVs),
                         replace = F)
DataTrainOnly <- DataTrainAndVs[DataTrainIndex,]
DataValidationOnly <- DataTrainAndVs[-DataTrainIndex,]

The rest of the data is randomly sampled and split into train and validation sets

For this example I'll use parameter defaults (because the issue behaves the same tuned or default)

MyModel <- randomForest(formula = RESPONSE ~.,
                        data = DataTrainOnly,
                        na.action = na.exclude)
#There are no NAs but last argument just cause

Training data information

Positive responses as a percent of total responses for the training data is 19.6%

OOB error rate I get is: 4.86%

Class error for 0 is: 0.01%

Class error for 1 is: 24.4%

MyPreds_Vs <- predict(object = MyModel,
                   newdata = DataValidOnly,
                   type = "class")
MyPreds_Test <- predict(object = MyModel,
                        newdata = DataTest,
                        type = "class")


Percent of positive responses is 19.1%

Sensitivity: 0.761

Specificity: 0.988


AUC: 0.874


Percent of positive responses is 21.1%


Specificity: 0.97



Why are the scores on the MyPreds_Test model so much lower than the MyPreds_Vs model?

One thought I had was something specific was happening in 2015, however when I randomly selected a different year for a hold out test set, lets say 2013, I saw the same issue. The validation data would perform well and the test data would not.

If anyone has had a similar issue, explanations of what's happening, or ways to investigate what is happening it would be greatly appreciated

  • 1
    $\begingroup$ I think you've verified that, in whichever domain you're working in, using a validation set separate in time is a good idea, because randomly sampling from the whole data set gives a result which is too optimistic. $\endgroup$ – Jonny Lomond Sep 9 at 13:18
  • $\begingroup$ @JonnyLomond Can you elaborate more? The reason I used an isolated year is ideally this model will be applied to data 6 months in the future so its temporally contiguous. Do you have any ideas why the performance is different? They are both unseen data.. $\endgroup$ – Jamalan Sep 9 at 13:22
  • $\begingroup$ What @JonnyLomond is saying is the problem is in "#Here I take 85% of the data from 2000 to 2014 to train with, #and 15% of the 2000-2014 data will be for validation and tuning". You should select 85 % of the years for training. $\endgroup$ – Roland Sep 9 at 13:36
  • $\begingroup$ @Roland Ah okay I see, so 85% of 14 years is about 12 years, so use 2000-2012 for test, then 2013-2014 for validation, and 2014-2015 for test? Although I can't understand why randomly sampling instead of stratified sampling would give better results? The data is unseen either way, and the data passing through the splitting criterion of the model would be subjected to the same splits. $\endgroup$ – Jamalan Sep 9 at 14:51
  • $\begingroup$ @Roland Also would stratifying in such a way imbalance the target classes and predictor classes? $\endgroup$ – Jamalan Sep 9 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.