Will I obtain seriously biased results if I use bootstrapping on a subsample of a larger dataset?

Rather than drawing 100 bootstrap samples from a dataset of 50 million + records, which could hog server resources, I'm thinking of first drawing a 5% random sample, w/ replacement, of records from the master dataset. Then build bootstrapped 95% confidence intervals for several statistics of interest using the 5% sample. So in essence I'll be bootstrapping from a bootstrap sample.

I'm working with health data which is clustered according to provider and episode of care, hence drawing a proper representative 5% sample will require some care. Some earlier advice has pointed me in the right direction ( Proper bootstrapping technique for clustered data? ).

  • 2
    $\begingroup$ The short answer is yes (the CIs will be "seriously biased"): the widths of the CIs from a 5% subsample will typically be about 4-5 times greater than the CIs from the full sample. Some creative thinking might be useful here. To engage that, you will need to tell us a little more, such as what you intend to use these CIs for and why you're analyzing such a large dataset as a whole rather than breaking it down into smaller, natural subsets for individual study. $\endgroup$ – whuber Nov 21 '12 at 20:41
  • 1
    $\begingroup$ @whuber While I'm not arguing with your point, it's very unintuitive to me - can you give some reasoning as to why this bias occurs? $\endgroup$ – Peter Flom Nov 21 '12 at 22:09
  • 2
    $\begingroup$ @Peter Notice, for instance, that the (usual) formula for the CI of a mean is proportional to $n^{-1/2}$ and to an estimated SD. The latter will be stable for such large samples, whereas $n^{-1/2}$ decreases with sample size. Although CI formulas for other estimators and other situations differ, the widths still typically show the $n^{-1/2}$ scaling. Intuitively, when you take a smaller sample you have to expect wider intervals, caeteris paribus. (Note that I am not referring to estimation bias, which is something altogether different.) $\endgroup$ – whuber Nov 21 '12 at 22:26
  • 1
    $\begingroup$ Ok, that makes sense and it's certainly clear for "regular" CI; I think something about the bootstrap situation threw my intuition off. $\endgroup$ – Peter Flom Nov 21 '12 at 22:52
  • $\begingroup$ Thanks gentlemen. I'm evaluating a baseline predictive model (built with 2007-2010 insurance claims) on more recent 2010-2011 and 2011-2012 data by scoring the baseline model parameters on the more recent time periods. We're trying to predict claim frequency for the entire episode of care based on freq. during the first 2-3 months. Sensitivity, specificity, and PPV statistics for will be compared to see if the baseline model holds up with more current data. 95% CIs for the three statistics would be helpful to measure precision, hence the bootstrapping. $\endgroup$ – RobertF Nov 24 '12 at 2:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.