I split my Dataset with 80% of the data for training and 20% for the test in the context of a binary classification task with a very unbalanced dataset.
On the training set I do a 3 folds StratifiedShuffleSplit cross-validation with a validation set size of 33% for each split, in order to have the same number of samples as in the test set. Here is my problem:
At the end of each split I get around 80% precision on both train / validation sets BUT only 65% precision on the test set. I have the same problem for the recall metric, it is way lower on the test set than on the validation subsets.
How can you explain to have such a difference between my validation scores at each split and my test scores ? I am a bit lost because validation sets are supposed to be data that my model doesn't encounter during the training as well as the test set.
Model used: LSTM with 15 input steps and 1 step prediction.