I'm designing a CNN model for a data mining competition in which we are provided with N sample of training data. We do not know the test size, but presumably it is from the same distribution as training set is from.

I obtained my best result while using 85% of data for training and 15% for validation to prevent the CNN from overfitting.

However different split sizes and data shuffling lead to different accuracies.

So, I wasn't sure if this type of training is reliable especially when we do not know the test size?


Your training/validation split doesn't depend on the size of outside test data. For a fair evaluation, you might also consider using some portion of your data as test set, e.g. 15-20 %. An example split can be 70/15/15. This way, you won't be able to use all of your data for the incoming test set; however, if you use your validation set for early stopping, it'll be valuable to have some other set to fairly evaluate your performance. Otherwise, you'll be using validation set twice, and your performance will be biased. Unfortunately, there is no golden ratio. There are only general rules of thumb, and practical choices.

| cite | improve this answer | |
  • $\begingroup$ I also learned that each time I shuffle the data before doing train/validation split I may get a different accuracy which varies between 90%-95%. Practically, should I use the model with the best validation accuracy? $\endgroup$ – Bob Dec 12 '19 at 1:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.