I have learned that resampling e.g. bootstrapping could give us better results for some problems. If we have a huge data set (millions of values) does it make sense to do some kind of resampling or these methods are only suitable when the data set is not that big?

  • $\begingroup$ Do you mean in your last sentence "or are these methods only suitable when the dataset is not that big?" $\endgroup$ Mar 14 '12 at 10:34

There is no reason why bootstrapping would be inappropriate with a large dataset, if inappropriate means deliver bad results because of the size. However, depending on how large the dataset and how complex the calculations that need to be done, there might be cost or efficiency problems.

  • $\begingroup$ Maybe philosophic question, but when we have a BIG dataset can bootstrapping give us anything more? Could you please mention some applications? We have a big collection of sensor data. $\endgroup$
    – James
    Mar 14 '12 at 10:42
  • $\begingroup$ Yes, I haven't given you a particularly good answer I know. Whether bootstrapping adds anything to traditional techniques does depend on a range of issues of which sample size might be one but more important are underlying distribution, adequacy of your model, how well your data answer your research questions, etc. $\endgroup$ Mar 14 '12 at 10:46
  • 2
    $\begingroup$ The correctness of this reply depends on the application. For a concrete example (large dataset, expensive parameter to evaluate) see mathematica.stackexchange.com/questions/2889/…. A solution was available there solely because the bootstrap samples could be efficiently calculated. $\endgroup$
    – whuber
    Mar 14 '12 at 16:51
  • 1
    $\begingroup$ Yes, @whuber is right, there is at least one potential reason it might be inappropriate which is cost/efficiency. $\endgroup$ Mar 14 '12 at 22:02

Since you have a huge dataset, you have the ability to validate many ideas on totally clean data. Keep a large portion at the side and go and try many ideas. You have a great "sit belt" that will prevent you from huge mistakes without having to worry about the dependencies of cross validation, overuse of specific samples and so. You are risking time, not performance.

As for the specific question of bootstrapping. Bootstrapping is usually used in order to reduce the variance of the model. You have a large dataset so the variance of the DATA should be low. However, it is possible that you use a classifier that its variance will be high so bootstrapping will help you. neural nets, classi cation and regression trees, and subset selection in linear regression are considered unstable (here , as self reference of Leo Breiman in Bagging Predictors)

Also, in many cases the dimensionality of the data is extremely high and your dataset doesn't cover it well enough.

When your dataset is imbalanced many classifiers might give you a prediction that is quite similar to the majority rule. Using bootstrapping might help but in such cases I would recommend using technique like boosting that deliberately forces the learner to focus at areas of interest.

Another benefit of bootstrapping is getting a confidence level for the prediction. However, since most classifier provide confidence this is usually not the reason to use bootstrapping.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.