Skip to main content
2 of 2
some clarification
Momo
  • 9.5k
  • 3
  • 49
  • 61

I don't know about "best" (which probably depends on what you use it for), but I use bootstrap validation to estimate error on new data the following way ( third way if you like):

  1. Draw a training set of N observations from the original data (of size N) with replacement.
  2. Fit the model to the training data.
  3. Evaluate the model on the out-of-bag (oob) samples

What is out of bag is not always clearly defined. Often it is all those observations that weren't part of the training set. More stricter would it be (I use it this way) to only have observations in the oob sample that have a realisation of the whole predictor vector that is not part of the training set (which is especially useful if you have many factors). Even stricter is to use an oob sample that contains only those observations who have a different realisation of the predictor variable on the predictors chosen in the model (especially useful if the model is found with some variable selection procedure, e.g. trees).

Then I usually repeat this a number k of times and aggregate results over the k-folds (mean or median or whatever statistic is handy). The model chosen this way can then be fitted to the overall data set (as in your option 2) to additionally gauge if there still is a tendency to overfit (the performance measure should be not too far off from the bootstrap samples).

If I have more models or a parameter grid or similar, I fit them all to each training set and evaluate them all on each oob sample. It is also possible to not use a training set twice, but for every model or tuning parameter combination to draw a new training/oob pair.

See e.g. The Design and Analysis of Benchmarking Experiments.

Momo
  • 9.5k
  • 3
  • 49
  • 61