3
$\begingroup$

General rule of thumb in splitting data in Machine Learning is in 3 parts training set, validation set and testing set. Well everybody knows that.

So, to try out the performance of different algorithms we try using dev set as test set and training set as the training set.

Now after finding correct hyperparameters of the model: Why don't we add validation and training sets together in order to obtain a bigger training set for the model to learn than the traditional model trained only on training set and validated on validation set?

Will not the new training + validation set provide more features? And therefore increasing accuracy of model after testing on the test set? Because more data implies more features to learn for the model and hence will increase the performance on test set as well.

Improve this question
$\endgroup$
3
  • $\begingroup$ If you mean to use more data by avoiding a large split in train/validate set, why not try cross-validation? Especially if you have the resources for leave-one-out, you are practically using all data. $\endgroup$
    – Frans Rodenburg
    Oct 10, 2017 at 6:31
  • $\begingroup$ I don't mean to give away the notion of split in train/validate set. But once we find optimal parameters, will it not be better to train on large examples (train+validate examples) rather than just train examples? $\endgroup$
    – SupposeXYZ
    Oct 10, 2017 at 9:14
  • $\begingroup$ Everybody knows that? Splitting of data when all 3 sample sizes are not huge is very inefficient statistically. Ever looked at the bootstrap for model validation? $\endgroup$
    – Frank Harrell
    Oct 15, 2017 at 11:43

1 Answer 1

Reset to default
1
$\begingroup$

The dataset we have can be in the thousands or millions.
Let's consider two cases :
Case 1 : Suppose that we have dataset of size 10,000. So we will divide the dataset in 60% 20% 20% rule to get 6000, 2000 and 2000 as size of training, dev and test set. In this case, adding validation set to training set makes difference to the feature learning capability of neural nets as more data means more feature exploitation.

Case 2: Suppose that we have dataset of size 10,000,000. So we will divide the dataset in 96% 2% 2% rule to get 9,600,000, 200,000 and 200,000 as size of training, dev and test set. In this case, the proportion of the examples that get added to training set are less compared to case 1. As our neural net has already learned from 9,600,000 examples for training adding extra 200,000 examples won't make as much difference in the accuracy.

Conclusion: If we have validation set in some proportion of training set and training set is less in size we can boost accuracy a bit by combining these two sets to let neural net learn a bit more on more examples. On other hand, if we have more amount of training set already it will make no difference to the accuracy by adding small portion of data to training set. It all matters on how we are dividing the train-dev-test distribution.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.