In order to properly estimate model prediction performance I use bootstrapping to estimate the confidence interval of the performance measure.

I perform a repeated random sampling to generate $B$ stratified samples $D_1, ..., D_B$ of the dataset $D$, with replacments.

  • Each sample consist of $N$ records, where $N$ is the number of records in the original dataset $D$.
  • "With replacements" means that a record can appear more than once in a sample.
  • "Stratified" - each sample $D_1, ..., D_B$ shell have the same proportions of positive samples $($ in case of binary label, $y=1)$.

Train-test partitioning: split each sample into two datasets: train and test (e.g., fit a model on $70\%$ of the data, evaluate on the rest $30\%$). Stratification - both train and test samples should be stratified as well.

A record that participated in the train shouldn't be used to validate the model. But, since each sample may be duplicated (due "with replacements") we need to make sure we don’t have overlapping in test. The solution I thought about is:

  1. Partition $D$ into two datasets: train set $D_{train}$ and test set $D_{test}$ (70%,30%) with stratification

  2. From $D_{train}$, do stratified bootstrap

  3. From $D_{test}$, stratified bootstrap

Any thoughts on this? is this the correct way to evaluate a model with bootstrapping?

  • $\begingroup$ I don't understand what is the problem with the replacement you comment us. Normally, we apply only bootstrapping to the training set. Then, we use the boostrap sample with N records (with duplcated) as train-test, and the rest (without duplicated) as testing-test. This is correct. $\endgroup$ Commented Feb 10, 2016 at 15:25
  • $\begingroup$ If you take "the rest" (i.e., any record that wasn't sampled for training) this doesn't promise you the test set will have the same proportion of positive sample as the train. $\endgroup$ Commented Feb 10, 2016 at 19:01


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.