1
$\begingroup$

Usually we partition the whole data-set into three parts: training data, validation data and test data. For the training data it is used to train the parameters of the variables in the model; and the validation set is for choosing or check the overfitting of the parameters; and for the test set it is only for testing the performance of a particular set of parameters. That's the measure of the generalization ability of our model.

And I also learned that if the data-set is too small, it is better to use all of the data as training data. But in such case how do we validate or test the model?

What are confusing me include:
1) When we divide our data-set into the three parts how can we do it in a right way in different scenarios? For instance if the data-set is too small it would be 1:0:0; but how small it should be to do that way? What if the size of our data is very large?
2) Should we just do the partition or we can process then separately afterwards? For instance if the test set is relatively small we can optimize those data. Do some data cleaning futher or something, which makes it different from the training data but more approximating to the real scenario if we try to apply the model.
3) I know that there is a very popular method named 10-fold crossvalidation, and it is also often utilized when we have small data-set. When to choose which method?

$\endgroup$
1
  • 1
    $\begingroup$ What do you mean by process the data afterwards? Your whole point 2) is quite obscure, would you mind clarifying it? $\endgroup$
    – gented
    Feb 27, 2019 at 23:55

1 Answer 1

1
$\begingroup$

If the dataset is too small, there is not enough data to use the method you are proposing.

For datasets of suitable size, there are various suggestions on how to partition. Normally you would have the training data as the largest subset. There seems to be no right answer on the relative sizes of the partitions. Hastie, Tibshirani, and Friendman suggest a 50/25/25 split for training, validation, and testing respectively.

The data should be cleaned prior to analysis. This must be done for the usual reasons, which may include aggregating categories so there are no sparse counts. Fiddling with data for other reasons--such as trying to get a poorly fitted model having a better fit-- is conceptually similar to p-value hacking. You'll get a great fit on your data, and the model will be useless for data coming in later.

You want to use a cross-validation method for the model. Your software should have this as an option within the code. The training/validation/testing data is randomly recombined so that the model results become independent of how the data is partitioned between the three subsets.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.