I'm building a binary classifier, where each record is a task, and the response variable is whether it was completed on time.
I'm using random forest My data set spans from 2000-2015
My hold out test set has far worse performance than my validation set
#Get last year of available data as test data DataTest <- Data %>% filter(DATE_YEAR == 2015) #The rest of the data is for training and validation DataTrainAndVs <- Data %>% filter(DATE_YEAR != 2015)
So I'm keeping the year 2015 as a hold out test set
#Here I take 85% of the data from 2000 to 2014 to train with, #and 15% of the 2000-2014 data will be for validation and tuning PRCNT_TR <- 0.85 DataTrainIndex <- sample(1:nrow(DataTrainAndVs), round(PRCNT_TR * nrow(DataTrainAndVs), replace = F) DataTrainOnly <- DataTrainAndVs[DataTrainIndex,] DataValidationOnly <- DataTrainAndVs[-DataTrainIndex,]
The rest of the data is randomly sampled and split into train and validation sets
For this example I'll use parameter defaults (because the issue behaves the same tuned or default)
MyModel <- randomForest(formula = RESPONSE ~., data = DataTrainOnly, na.action = na.exclude) #There are no NAs but last argument just cause
Training data information
Positive responses as a percent of total responses for the training data is 19.6%
OOB error rate I get is: 4.86%
Class error for 0 is: 0.01%
Class error for 1 is: 24.4%
MyPreds_Vs <- predict(object = MyModel, newdata = DataValidOnly, type = "class") MyPreds_Test <- predict(object = MyModel, newdata = DataTest, type = "class")
Percent of positive responses is 19.1%
Percent of positive responses is 21.1%
Why are the scores on the MyPreds_Test model so much lower than the MyPreds_Vs model?
One thought I had was something specific was happening in 2015, however when I randomly selected a different year for a hold out test set, lets say 2013, I saw the same issue. The validation data would perform well and the test data would not.
If anyone has had a similar issue, explanations of what's happening, or ways to investigate what is happening it would be greatly appreciated