17
$\begingroup$

When using bootstrapping for model evaluation, I always thought the out-of-bag samples were directly used as a test set. However, this appears not to be the case for the deprecated scikit-learn Bootstrap approach, which seems to build the test set from drawing with replacement from the out-of-bag data subset. What is the statistical reasoning behind this? Are there specific scenarios where this technique is better than just evaluating on the out-of-bag-sample or vice versa?

$\endgroup$
3
  • $\begingroup$ cross-validation? resampling? $\endgroup$ May 5 '15 at 16:10
  • $\begingroup$ If you're referring to the tag, that was not added by me. I'm bootstrapping as an alternative to CV, actually. $\endgroup$
    – gibberfish
    May 7 '15 at 12:28
  • $\begingroup$ I think you misunderstand. In cross validation they are going to repeat the process several times on a subset of the data and look at the variation in outcomes or model parameters. In that case the parameter variation itself informs quality of fit, and not necessarily the OOB error. I am not familiar with the particular library so I just left a comment with the idea. $\endgroup$ May 7 '15 at 12:34
3
$\begingroup$

Bootstrap samples are used to evaluate the performance of the algorithm by many iterations. While doing so, the performance on randomly changed sets is evaluated.

In contrast when doing for example 10 Fold Cross Validation you are performing only 10 iterations on different train and test data sets.

Now when your sample size is small, lets say $n=20$ and the number of bootstrap iterations is high, lets choose $i=10,000$, and you do not resample your test data as you do with your train data set you will have situations where your algorithm sees the same or very similar test more than one time. A situation you originally wanted to avoid by using bootstrap.

The link you postet is down, so I added the description of the function in the current (0.14) version of sklearn

Description of the method

Random sampling with replacement cross-validation iterator Provides train/test indices to split data in train test sets while resampling the input n_iter times: each time a new random split of the data is performed and then samples are drawn (with replacement) on each side of the split to build the training and test sets. Note: contrary to other cross-validation strategies, bootstrapping will allow some samples to occur several times in each splits. However a sample that occurs in the train split will never occur in the test split and vice-versa. If you want each sample to occur at most once you should probably use ShuffleSplit cross validation instead.

$\endgroup$
3
$\begingroup$

Perhaps you were onto something. It seems others pulled at that same thread and Bootstrap was deprecated in favor of more intentional use of the resample method with the tried and true sklearn.cross_validation approaches like StratifiedKFold.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.