1
$\begingroup$

I'm a rather novice to statistics, so my terminology here might not be totally correct. But I will do my best to explain my question clearly.

My question is about understanding whether it is making sense to use bootstrapping method to calculate a confidence interval for a certain parameter of a population.

  1. In case of estimating a population mean, what I studied is that sample mean is my best point estimate. And with the central limit theorem, the center point of a "sample mean distribution" should be same as that of a population if the sample size and the number of resampling are big enough. And the sample distribution of mean is normally distributed and from here, I can define a confidence interval of the sample mean distribution. And the population mean is expected to be within this confidence interval. A lot of online materials explain this case.

  2. But let's assume that there is an imaginary statistics "Foo" and I want to calculate a population confidence interval of "Foo" from the bootstrapped sample distribution. But we don't know the relation between the sample distribution of the "Foo" and the population parameter "Foo", not like the mean statistics explained above. Then, it makes no sense to me to use bootstrapping method to estimate the population parameter "Foo". The best I can do is, calculating sample distribution of the "Foo". Is my understanding here correct?

  3. If my understanding in the paragraph 2 is correct, then what are the practical usages of bootstrapping method?

Thanks.

$\endgroup$
1
$\begingroup$

Background. The traditional way to get a 95% CI for the population standard deviation of a normal distribution (with mean unknown) is to pivot the the quantity $\frac{(n-1)S^2}{\sigma^2} \sim\mathsf{Chisq}(\nu=n-1)$ to obtain the 95% CI for $\sigma^2$ of the form $\left(\frac{(n-1)S^2}{U},\frac{(n-1)S^2}{L}\right),$ where $L$ and $U$ cut probability $0.025$ from the lower and upper tails, respectively, of $\mathsf{Chisq}(n-1).$ to get a 95% CI for $\sigma^2.$ And then take square roots of endpoints to get a 95% CI for $\sigma.$

Thus if $X_1, \dots, X_{20}$ is a random sample from $\mathsf{Norm}(\mu=50,\sigma=7),$ which has $S^2 = 7.62,$ then we can find a 95% CI $(6.37,9.50)$ for $\sigma,$ as follows. [Sampling and computation in R.]

set.seed(2021)
x = rnorm(20, 50, 7)
s = sd(x);  s
[1] 7.621391
sqrt(49*s^2/qchisq(c(.975,.025), 49))
[1] 6.366407 9.497269

Simple quantile bootstrap. Now suppose we had the vector x without knowing the distribution from which it was sampled, and want to get a 95% nonparametric bootstrap CI for $\sigma$ using a simple quantile method based on sample standard deviations.

We get the CI $(4.90,9.43).$ It contains $\sigma=7$ (near the center), but it is longer than the CI that uses normal distribution theory. The increased length is not surprising because we have (and use) less information about the source of the data.

set.seed(201)
s = replicate(2000, sd(sample(x,20,rep=T)))
quantile(s, c(.025, .975))
     2.5%    97.5% 
4.898676 9.431256 

There are many methods of nonparametric bootstrapping for CIs that might have been used, some of them arguably better, but the quantile bootstrap above is one of the simpler methods.

Now suppose we have a random sample y of size $n = 100$ with sample standard deviation $S = 1.063.$ We know only that it is from a continuous distribution for which the population standard deviation $\sigma$ exists.

summary(y); length(y);  s=sd(y);  s
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.4955  1.3131  1.8799  2.0798  2.5453  6.6592 
[1] 100      # sample size
[1] 1.063716 # sample sd.

enter image description here

A 95% nonparametric bootstrap CI using the simple quantile method is $(0.82,1.30).$ [Because I simulated y from the distribution $\mathsf{Beta}(\mathrm{shape}=4,\mathsf{rate}=2),$ I know that the population $\sigma=1.$ As far as I know, there is no really simple parametric CI for $\sigma.]$

set.seed(121)
sd = replicate(2000, sd(sample(y,100,rep=T)))
quantile(sd, c(.025, .975))
     2.5%     97.5% 
0.8221678 1.3001359 

Pivotal Bootstrap. A somewhat different kind of CI for $\sigma$ might be to say that if I knew the distribution theory I could find $L$ and $U$ such that $P(L < S - \sigma < U)= 0.95,$ I could pivot to get a 95% CI of the form $(S-U, S-L)$ for $\sigma.$

Not knowing, the distribution theory I use bootstrapping to estimate $L$ and $U$ by $L^*$ and $U^*,$ respectively, temporarily using the observed $S_{obs}$ as a proxy for unknown $\sigma.$

In the R code, I use the suffix .re to denote re-sampled quantities. In the last step s.obs returns to its role as the observed sample SD. This style of bootstrap CI gives the interval $(0.84, 1.29),$ which is not much different numerically from the simple quantile bootstrap CI above. However, for markedly skewed distributions, the pivotal method may have advantages.

set.seed(1234)
s.obs = sd(y)
d.re = replicate(2000, sd(sample(y,100,rep=T))-s.obs)
UL = quantile(d.re, c(.975,.025))
s.obs - UL
    97.5%      2.5% 
0.8444399 1.2913911 

Note: Sample y was simulated in R as follows:

set.seed(1234)
y = rgamma(100, 4, 2)
sd(y)
[1] 1.063716
$\endgroup$
8
  • $\begingroup$ Dear BruceET, Thanks for your answer. For the background part, that's because we know that the sample distribution is chi-square distributed with a certain DOF. A standard deviation from one sample should belong to this distribution and that's how we calculate the confidence interval. $\endgroup$ – Bicycle-riding Dog Feb 1 at 12:24
  • 1
    $\begingroup$ For the other your bootstrap examples, the distributions you calculated are all about the distribution of one sample you took or distribution of statistics. (I'm not sure which one is correct). And we don't know the relation between the population parameter and sample statistics. Then how can we define a confidence interval out of sample distribution? Would you please explain this one further if you don't mind? $\endgroup$ – Bicycle-riding Dog Feb 1 at 12:24
  • $\begingroup$ In a practical problem, we may believe a distribution of weights, scores, costs, etc. is normal or nearly normal, but we may not know the values of the population mean $\mu$ or the population standard deviation $\sigma.$ Using a random sample of $n$ observations randomly chosen from the population we could compute the sample mean $\bar X=\frac 1 n\sum_{i=1}^n X_i$ and the sample standard deviation $S = \sqrt{\frac{1}{n-1}\sum_{i=1}^n (X_i-\bar X)^2}.$ Then we might use $\bar X$ as an estimate of $\mu$ and $S$ as an estimate of $\sigma.$ CIs are used to show how precise the estimates may be. $\endgroup$ – BruceET Feb 1 at 21:42
  • $\begingroup$ Thanks for a reply, BruceET. But that is exactly the central limit theorem I mentioned in the opening question about mean value. And you nicely described the relation between sample standard deviation and population parameter with chi-square distribution in the other answer. But what about statistics of bootstrapped sampling distribution? I think that to estimate confidence interval for a certain parameter, we need to know a relation between bootstrapped sampling distribution and the parameter. This is what I'm missing. $\endgroup$ – Bicycle-riding Dog Feb 2 at 8:02
  • $\begingroup$ Consider my example with data from a gamma distribution. Even though, with the usual parameterizations of the gamma distribution, the SD $\sigma$ of the distribution is not a 'parameter', there is no doubt that a gamma distribution has a SD. There may be applications in which it is useful to estimate $\sigma$ and to have a CI for $\sigma$ and the sample SD $S$ seems a good way to estimate $\sigma.$ // One of the advantages of bootstrapping is its ability to find such CIs without knowing the distribution of the sample SD $S.$ That's one of the the reasons I chose a gamma sample for my example. $\endgroup$ – BruceET Feb 2 at 8:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.