9
$\begingroup$

Provided a sample size "N" that I plan on using to forecast data. What are some of the ways to subdivide the data so that I use some of it to establish a model, and the remainder data to validate the model?

I know there is no black and white answer to this, but it would be interesting to know some "rules of thumb" or usually used ratios. I know back at university, one of our professors used to say model on 60% and validate on 40%.

$\endgroup$
7
$\begingroup$

Well as you said there is no black and white answer. I generally don't divide the data in 2 parts but use methods like k-fold cross validation instead.

In k-fold cross validation you divide your data randomly into k parts and fit your model on k-1 parts and test the errors on the left out part. You repeat the process k times leaving each part out of fitting one by one. You can take the mean error from each of the k iterations as an indication of the model error. This works really well if you want to compare the predictive power of different models.

One extreme form of k-fold cross validation is the generalized cross validation where you just leave out one data point for testing and fit the model to all the remaining points. Then repeat the process n times leaving out each data point one by one. I generally prefer k-fold cross validation over the generalized cross validation ... just a personal choice

$\endgroup$
  • 2
    $\begingroup$ CV using full set for model selection, huh? It's a common error (still even Wikipedia mentions it), because it is a hidden overfit. You need to make a higher level CV or leave some test to do this right. $\endgroup$ – user88 Jul 29 '10 at 8:00
5
$\begingroup$

1:10 test:train ratio is popular because it looks round, 1:9 is popular because of 10-fold CV, 1:2 is popular because it is also round and reassembles bootstrap. Sometimes one gets a test from some data-specific criteria, for instance last year for testing, years before for training.

The general rule is such: the train must be large enough to so the accuracy won't drop significantly, and the test must be large enough to silence random fluctuations.

Still I prefer CV, since it gives you also a distribution of error.

$\endgroup$
4
$\begingroup$

It really depends on the amount of data you have, the specific cost of methods and how exactly you want your result to be.

Some examples:

If you have little data, you probably want to use cross-validation (k-fold, leave-one-out, etc.) Your model will probably not take much resources to train and test anyway. It are good ways to get the most out of your data

You have a lot of data: you probably want to take a reasonably large test-set, ensuring that there will be little possibility that some strange samples will give to much variance to your results. How much data you should take? It depends completely on your data and model. In speech recognition for example, if you would take too much data (let's say 3000 sentences), your experiments would take days, as a realtime factor of 7-10 is common. If you would take too little, it is too much dependent on the speakers that you are choosing (which are not allowed in the training set).

Remember also, in a lot of cases it is good to have a validation/development set too!

$\endgroup$
4
$\begingroup$

As an extension on the k-fold answer, the "usual" choice of k is either 5 or 10. The leave-one-out method has a tendency to produce models that are too conservative. FYI, here is a reference on that fact:

Shao, J. (1993), Linear Model Selection by Cross-Validation, Journal of the American Statistical Association, Vol. 88, No. 422, pp. 486-494

$\endgroup$
  • $\begingroup$ Have you even read this paper? Nevertheless it is works only for linear models (even the title shows it!) it is about asymptotic behavior for infinite number of objects. 100 is way not enough. $\endgroup$ – user88 Jul 29 '10 at 7:49
  • 1
    $\begingroup$ And I wish you luck making 10-fold cross validation on set with 9 objects. $\endgroup$ – user88 Jul 29 '10 at 7:50
  • $\begingroup$ @mbq: I say the "usual" choice. Does not mean every choice $\endgroup$ – Albort Aug 23 '10 at 2:52
  • $\begingroup$ @mbq: I have read the paper; Shao reports on a simulation study with only 40 observations, and shows that LOOCV underperforms the Monte-Carlo CV except in the case where no subselection is appropriate (the full feature set is optimal). 100 is way more than enough, at least for subset selection in linear models. $\endgroup$ – shabbychef Sep 2 '10 at 23:14
  • $\begingroup$ @shabbychef You've got me here; the second argument in my first comment is of course a junk, I had some other works in mind and overgeneralized. Nevertheless, I will still argue that Shao's paper is not a good reference for general "LOO fails for large N" since its scope is reduced to linear models. $\endgroup$ – user88 Sep 3 '10 at 6:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.