43
$\begingroup$

I've been looking at numerous questions on this site regarding bootstrapping and confidence intervals, but I'm still confused. Part of the reason for my confusion is probably that I'm not advanced enough in my statistics knowledge to understand a lot of the answers. I'm about half-way through an introductory statistics course and my math level is only about mid-Algebra II, so anything past that level just confuses me. If one of the knowledgeable people on this site could explain this issue at my level it would be extremely helpful.

We were learning in class how to take resamples using the bootstrap method and use those to build up a confidence interval for some statistic we'd like to measure. So for example, say we take a sample from a large population and find that 40% say they'll vote for Candidate A. We assume that this sample is a pretty accurate reflection of the original population, in which case we can take resamples from it to discover something about the population. So we take resamples and find (using a 95% confidence level) that the resulting confidence interval ranges from 35% to 45%.

My question is, what does this confidence interval actually mean?

I keep reading that there's a difference between (Frequentist) Confidence Intervals and (Bayesian) Credible Intervals. If I understood correctly, a credible interval would say that there's a 95% chance that in our situation the true parameter is within the given interval (35%-45%), while a confidence interval would say that there's a 95% that in this type of situation (but not necessarily in our situation specifically) the method we're using would accurately report that the true parameter is within the given interval.

Assuming this definition is correct, my question is: What's the "true parameter" that we're talking about when using confidence intervals built up using the bootstrap method? Are we referring to (a) the true parameter of the original population, or (b) the true parameter of the sample? If (a), then we'd be saying that 95% of the time the bootstrap method will accurately report true statements about the original population. But how could we possibly know that? Doesn't the whole bootstrap method rest on the assumption that the original sample is an accurate reflection of the population it was taken from? If (b) then I don't understand the meaning of the confidence interval at all. Don't we already know the true parameter of the sample? It's a straightforward measurement!

I discussed this with my teacher and she was quite helpful. But I'm still confused.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
30
$\begingroup$

If the bootstrapping procedure and the formation of the confidence interval were performed correctly, it means the same as any other confidence interval. From a frequentist perspective, a 95% CI implies that if the entire study were repeated identically ad infinitum, 95% of such confidence intervals formed in this manner will include the true value. Of course, in your study, or in any given individual study, the confidence interval either will include the true value or not, but you won't know which. To understand these ideas further, it may help you to read my answer here: Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?

Regarding your further questions, the 'true value' refers to the actual parameter of the relevant population. (Samples don't have parameters, they have statistics; e.g., the sample mean, $\bar x$, is a sample statistic, but the population mean, $\mu$, is a population parameter.) As to how we know this, in practice we don't. You are correct that we are relying on some assumptions--we always are. If those assumptions are correct, it can be proven that the properties hold. This was the point of Efron's work back in the late 1970's and early 1980's, but the math is difficult for most people to follow. For a somewhat mathematical explanation of the bootstrap, see @StasK's answer here: Explaining to laypeople why bootstrapping works . For a quick demonstration short of the math, consider the following simulation using R: 

# a function to perform bootstrapping
boot.mean.sampling.distribution = function(raw.data, B=1000){
  # this function will take 1,000 (by default) bootsamples calculate the mean of 
  # each one, store it, & return the bootstrapped sampling distribution of the mean

  boot.dist = vector(length=B)     # this will store the means
  N         = length(raw.data)     # this is the N from your data
  for(i in 1:B){
    boot.sample  = sample(x=raw.data, size=N, replace=TRUE)
    boot.dist[i] = mean(boot.sample)
  }
  boot.dist = sort(boot.dist)
  return(boot.dist)
}

# simulate bootstrapped CI from a population w/ true mean = 0 on each pass through
# the loop, we will get a sample of data from the population, get the bootstrapped 
# sampling distribution of the mean, & see if the population mean is included in the
# 95% confidence interval implied by that sampling distribution

set.seed(00)                       # this makes the simulation reproducible
includes = vector(length=1000)     # this will store our results
for(i in 1:1000){
  sim.data    = rnorm(100, mean=0, sd=1)
  boot.dist   = boot.mean.sampling.distribution(raw.data=sim.data)
  includes[i] = boot.dist[25]<0 & 0<boot.dist[976]
}
mean(includes)     # this tells us the % of CIs that included the true mean
[1] 0.952
Improve this answer
$\endgroup$
9
  • $\begingroup$ Which particular assumptions are we relying on? $\endgroup$
    – iarwain
    Commented Jan 30, 2014 at 2:02
  • 2
    $\begingroup$ Thanks. I think I found what I was looking for in the second answer to that thread: "Remember that we are not using the means of the bootstrap samples to estimate the population mean, we use the sample mean for that (or whatever the statistic of interest is). But we are using the bootstrap samples to estimate properties (spread, bias) of the sampling proccess. And using sampling from a know population (that we hope is representative of the population of interest) to learn the effects of sampling makes sense and is much less circular." ... $\endgroup$
    – iarwain
    Commented Jan 30, 2014 at 17:14
  • 1
    $\begingroup$ ... In other words, all the CI is telling us is that in a population roughly similar to ours we would expect 95% of samples taken from that population to reflect the true value +/- the margin of error. So all we're doing is giving a very rough clue - though perhaps the best clue we have - to around how close our sample statistic might be to the true population parameter. If so, then it sounds like we shouldn't take the exact numbers in the CI too seriously - they just mean something like, "the sample statistic is probably roughly accurate to probably roughly this degree." Did I get that right? $\endgroup$
    – iarwain
    Commented Jan 30, 2014 at 17:42
  • 1
    $\begingroup$ That's essentially correct. A CI gives us a sense of the precision of our estimate, but we never know if our actual (realized) CI does contain the true value. The primary assumption is that our data are representative of the population of interest. Note that neither of these are particular to bootstrapped CIs, you have the same interpretation & assumption in a CI calculated via asymptotic theory. $\endgroup$
    – gung - Reinstate Monica
    Commented Jan 30, 2014 at 18:05
  • 1
    $\begingroup$ This is an excellent explanation. I would add only that the "true value" is at times an artifact of the study design. In polling for political candidates, stratified samples give much more precise and reliable estimates than would a random sample. The cost is a risk of oversampling the wrong group by design. In that case, the 95% CI is centered on the correct value, the one which is achieved by replicating the study ad infinitum, but that value is not the other sense of a true parameter: the parameter we wanted to estimate. This is why study design and inference are intrinsically linked. $\endgroup$
    – AdamO
    Commented Dec 13, 2017 at 17:12
0
$\begingroup$

What you are saying is that there is no need to find confidence interval from bootstrapped resamples. If you are satisfied with the statistic (sample mean or sample proportion) obtained from bootstrapped resamples, do not find any confidence interval and so, no question of interpretation. But if you are not satisfied with the statistic obtained from bootstrapped resamples or satisfied but still want to find the confidence interval, then the interpretation for such confidence interval is same as any other confidence interval. It's because when your bootstrapped resamples are exactly representing (or assumed to be so) the original population, then where is the need of confidence interval? The statistic from the bootstrapped resamples is the original population parameter itself but when you do not consider the statistic as the original population parameter, then there is a need to find the confidence interval. So, it's all about how you consider. Let's say you calculated 95% confidence interval from bootstrapped resamples. Now the interpretation is: "95% of the times, this bootstrap method accurately results in a confidence interval containing the true population parameter".

(This is what I think. Correct me if there are any mistakes).

Improve this answer
$\endgroup$
-1
$\begingroup$

We are referring to the true parameter of the original population. It is possible to do this assuming that the data were drawn randomly from the original population -- in that case, there are mathematical arguments showing that the bootstrap procedures will give a valid confidence interval, at least as the size of the dataset becomes sufficiently large.

Improve this answer
$\endgroup$
3
  • $\begingroup$ So it sounds like in order to understand why it works I'll need to know enough math to follow the mathematical proofs. Is that correct? $\endgroup$
    – iarwain
    Commented Jan 30, 2014 at 0:04
  • $\begingroup$ I think so (I am not familiar with the proofs) $\endgroup$
    – Gareth
    Commented Jan 30, 2014 at 1:49
  • $\begingroup$ Intuitively though, you can see that as the sample size gets large, the sample starts to look a lot like the population. For example, say I take 1 million samples from a normal distribution with given mean and variance. Call this sample X. A random sample (with replacement) drawn from X looks a lot like a random sample drawn from the original distribution. I think this is the basic idea of why it works. $\endgroup$
    – Gareth
    Commented Jan 30, 2014 at 1:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.