2
$\begingroup$

I'm somewhat new to understanding statistics meaningfully, and the bootstrap in particular.

Let's say you're doing something (polling, running experiment, whatever), and can afford to get 1000 samples.

From what I understand, before the bootstrap method existed (> 25-30 years ago), you'd be better off with, say, 10 sets of 100 samples each and calculating your statistics based on sampling distributions. (As an aside, is there a good way to calculate the ratio of sets/samples? I'm know that you get a smaller standard error when n is larger, but don't know if there is a "sweet spot", and from my rather limited understanding, it's preferable to have n > 30 to avoid messing around with t statistics).

But now that bootstrapping exists, my understanding is that you get one set of all 1000 samples (since this will be a better proxy for real poplulation than 100 would be), and do the bootstrapping resampling magic based on that.

Is this correct? Or is best practice some hybrid of the two? Or does it ultimately depend on what the actual activity is?

Thanks!

Improve this question
$\endgroup$
5
  • $\begingroup$ Your characterization of statistical methodology "before the bootstrap existed" is unclear and does not seem to correspond to anything that really occurred. This makes your question difficult to understand and perhaps impossible to answer. Could you explain what you are talking about or at least provide a reference to it? $\endgroup$
    – whuber ♦
    Dec 18, 2013 at 16:57
  • $\begingroup$ The bootstrap methodology is only about 25-30 years old, right? So I mean, what statisticians did > 25-30 years ago... $\endgroup$
    – retnuH
    Dec 18, 2013 at 17:34
  • $\begingroup$ Okay I see the paper was written in 1979, and from what understand, it took a few years before it became common practice and people had the computing resources to use it, so 25-30 years is probably about right... $\endgroup$
    – retnuH
    Dec 18, 2013 at 17:38
  • $\begingroup$ I am still mystified: your description does not correspond to anything recognizable in standard statistical practice of any era. I simply cannot tell what procedure you are attempting to contrast with bootstrapping. $\endgroup$
    – whuber ♦
    Dec 18, 2013 at 18:06
  • $\begingroup$ Okay between your mystification and AdamO's answer, I think I see what I've got wrong in my head. His first sentence pretty much nails it on the head, I think! $\endgroup$
    – retnuH
    Dec 18, 2013 at 19:59

1 Answer 1

Reset to default
4
$\begingroup$

You may have a confused understanding of the differences between bootstraps, the interpretation of frequentist statistics, and stratified sampling.

In particular, sampling 100 PSUs from 10 distinct clusters is beneficial when you have pre-selected such clusters based on either distribution of cluster level confounding variables or representative convenience sampling. (e.g. sampling 100 kids from 10 school districts rather than 1000 kids randomly from among 400 school districts). This is stratified sampling and is used to 1. alleviate confounding, 2. improve response rate, 3. make survey methods more affordable.

One can conceive, in general, if a random experiment were conducted in which 100 individuals were sampled and if this process were repeated 10 times, we might be able to infer the distribution of the test statistics from each replication based on confidence intervals generated from a single replication. This is the concept of inference in the frequentist sense, and the interpretation of the confidence interval is the coverage rate for an infinite number replications of the experiment performed at the same sample size.

The bootstrap is very closely related to the frequentist interpretation of test statistics and their standard error estimates. That's why the bootstrap (often) gives consistent estimates of these exact SEs in the presence of violations of model based assumptions. This is because we abandon any asymptotic or probability model based assumptions about the data generating process. We instead conceive of the empirical distribution function (which assigns probability 1/n to each of the n observations in our analysis) and calculate statistics by quite literally replicating our experiment an infinite number of times (we don't actually do this, but regular estimates converge to stable values with a few 1000 replications very easily).

So returning back to the concept of sampling 1,000 individuals. In a frequentist analysis, you can calculate estimates, standard errors, and perform inference in the usual way. The interpretation of those values follow the usual format. As far as the validity of the results, there are usually limit theorems which assure us that, under certain regularity conditions, the exact distribution of the test statistic should be very closely approximated by the limiting distribution of the test statistic. The bootstrap eschews similar limit theorems (although it has its own set of limit theorems, ref Glivenko Cantelli), and estimates the exact distribution of the test statistic.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for that, I can see how there were a few things jumbled in my head! This definitely gives me food for thought... $\endgroup$
    – retnuH
    Dec 18, 2013 at 20:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.