I split my Dataset with 80% of the data for training and 20% for the test in the context of a binary classification task with a very unbalanced dataset.

On the training set I do a 3 folds StratifiedShuffleSplit cross-validation with a validation set size of 33% for each split, in order to have the same number of samples as in the test set. Here is my problem:

At the end of each split I get around 80% precision on both train / validation sets BUT only 65% precision on the test set. I have the same problem for the recall metric, it is way lower on the test set than on the validation subsets.

How can you explain to have such a difference between my validation scores at each split and my test scores ? I am a bit lost because validation sets are supposed to be data that my model doesn't encounter during the training as well as the test set.

Model used: LSTM with 15 input steps and 1 step prediction.

  • $\begingroup$ Do your test and validation sets come from the same distribution- in other words, do they have similar data, or does one set have unique examples as opposed to the other? $\endgroup$ – Shan S Jul 3 at 9:49
  • $\begingroup$ I randomly pick 20% of the data for testing among all data available, and then the Stratified Shuffle Split has the specificity of keeping the same proportion of classes between each split so that data in each split are conform to the original dataset. $\endgroup$ – Thomas Hustache Jul 3 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy