Validation Set Accuracy Significantly Higher than Hold out Test Set

Question

I'm building a binary classifier, where each record is a task, and the response variable is whether it was completed on time.

I'm using random forest My data set spans from 2000-2015

My hold out test set has far worse performance than my validation set

#Get last year of available data as test data
DataTest <- Data %>% filter(DATE_YEAR == 2015)
#The rest of the data is for training and validation
DataTrainAndVs <- Data %>% filter(DATE_YEAR != 2015)

So I'm keeping the year 2015 as a hold out test set

#Here I take 85% of the data from 2000 to 2014 to train with, 
#and 15% of the 2000-2014 data will be for validation and tuning
PRCNT_TR <- 0.85
DataTrainIndex <- sample(1:nrow(DataTrainAndVs), 
                         round(PRCNT_TR * nrow(DataTrainAndVs),
                         replace = F)
DataTrainOnly <- DataTrainAndVs[DataTrainIndex,]
DataValidationOnly <- DataTrainAndVs[-DataTrainIndex,]

The rest of the data is randomly sampled and split into train and validation sets

For this example I'll use parameter defaults (because the issue behaves the same tuned or default)

MyModel <- randomForest(formula = RESPONSE ~.,
                        data = DataTrainOnly,
                        na.action = na.exclude)
#There are no NAs but last argument just cause

Training data information

Positive responses as a percent of total responses for the training data is 19.6%

OOB error rate I get is: 4.86%

Class error for 0 is: 0.01%

Class error for 1 is: 24.4%

MyPreds_Vs <- predict(object = MyModel,
                   newdata = DataValidOnly,
                   type = "class")
MyPreds_Test <- predict(object = MyModel,
                        newdata = DataTest,
                        type = "class")

MyPreds_Vs

Percent of positive responses is 19.1%

Sensitivity: 0.761

Specificity: 0.988

F1:0.835

AUC: 0.874

MyPreds_Test

Percent of positive responses is 21.1%

Sensitivity:0.14

Specificity: 0.97

F1:0.23

AUC:0.55

Why are the scores on the MyPreds_Test model so much lower than the MyPreds_Vs model?

One thought I had was something specific was happening in 2015, however when I randomly selected a different year for a hold out test set, lets say 2013, I saw the same issue. The validation data would perform well and the test data would not.

If anyone has had a similar issue, explanations of what's happening, or ways to investigate what is happening it would be greatly appreciated