Concerning the number of samples in the dataset: Is there a rule of thumb considering when we should do cross-validation and when we should split our data into train, development and test sets?

Let's say I have a small dataset with almost 1000 samples. Does it make sense to split it into train, development and test sets, or I should rather do cross-validation?

  • $\begingroup$ I think it has to do with how well your model is fitting the data. You have to keep some "test" (totally unseen) data for any case, but if you have too many covariates then only one training wouldn't suffice, you have to go with cross-validation. In the opposite case though, you can go with either test-train split or cross-validation. $\endgroup$ – Ujjwal Kumar Jan 18 '17 at 13:16

We use Cross-validation when we want to check how well our model performs on "future unseen(Points not in train and test dataset)" data. i.e How well our algorithm can be generalized.

Normally, the "Random Splitting" is,

                          Training = 60%
                          Cross-Validation = 20%
                          Test = 20%

Normally, the "Time based splitting" is, If I am using 90 days time window then,

                           Training = data from day 1 to day 60
                           Cross-Validation = data from day 61 to day 75
                           Test = data from day 76 to 90
| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.