4
$\begingroup$

In trying to understand the bootstrap method, I have taken a sample of 11 observations from a Poisson distribution with a mean of two. I have obtained the following sample:

11 observations obtained from a Poisson distribution of mean two

Assuming I do not know the true distribution, I want to use bootstrap to estimate the confidence interval of the mean. I take one million bootstrap samples and compute the mean. This leads to the following bootstrap sampling distribution of the mean:

Average of one million bootstrap samples from the above set

But this does not look like the true sampling distribution of the mean, obtained by sampling one million samples from the Poisson distribution of mean two:

Average of one million samples taken from the Poisson distribution with mean two

How should I understand this? Does the bootstrap not work in this case? I am somewhat confused, as I thought the bootstrap would be able to come close to the true sampling distribution.

$\endgroup$
6
  • 4
    $\begingroup$ You are comparing two different distributions: your sample doesn't look exactly like a Poisson distribution, does it? $\endgroup$
    – whuber
    Commented Nov 22, 2023 at 15:45
  • $\begingroup$ Do you just chalk it up as bad luck then? As the resulting estimates are somewhat poor $\endgroup$
    – hhh3
    Commented Nov 22, 2023 at 18:26
  • 5
    $\begingroup$ This is ordinary luck. The sample is consistent with a Poisson distribution; there's nothing wrong there. Perhaps the real lesson, as given by John Snow in the answer, is that the justification for the bootstrap is asymptotic, relying on having a sufficiently large sample. Using the bootstrap with a small sample is risky -- just as any nonparametric procedure will be when applied to small samples. $\endgroup$
    – whuber
    Commented Nov 22, 2023 at 19:08
  • $\begingroup$ Bootstrapping is an approximation. There will always be an error (but that error can be made as small as you like by increasing the sample size). $\endgroup$ Commented Nov 23, 2023 at 9:09
  • $\begingroup$ Relating to that last comment is the question Why are hypothesis tests still used when we have the bootstrap and central limit theorem? to which the answer is that bootstrapping is not an exact method and an approximation. Like in your question the bootstrapping sample distribution is not equivalent to the true sample distribution. $\endgroup$ Commented Nov 23, 2023 at 9:16

2 Answers 2

22
$\begingroup$

The bootstrap validity hinges on the asymptotic theory. 11 observations sounds like a very small sample. Try to simulate a sample of size n=100 or n=1000 and then bootstrap it. Does it look better? What if you make n smaller, does it start looking worse?

P.S. I was going to write a comment, but could not because I do not have 50 reputation points.

$\endgroup$
2
  • 9
    $\begingroup$ Welcome to CV, John. I am sure you will have enough reputation soon to do whatever you like ;-). $\endgroup$
    – whuber
    Commented Nov 22, 2023 at 19:08
  • 6
    $\begingroup$ I think this is perfectly valid as an answer anyway, rather than a comment. $\endgroup$ Commented Nov 23, 2023 at 1:47
14
$\begingroup$

Expanding on the answer by John Snow (+1), the gist behind the bootstrap is that, if we can't go back and sample from the original distribution, the next-best option is to use the empirical distribution and sample from it. This makes the assumption that, loosely speaking, the empirical distribution is a good representation of the original distribution.

Your empirical distribution diverges from a $\text{Poisson}(2)$ distribution in a number of ways. Let $X$ be a random variable distributed as your empirical distribution.

  1. $P(X>6) = 0$

  2. $P(6>X>3) = 0$

For $Y\sim\text{Poisson}(2)$, it is the case that $P(Y>6) = 0.0166$ and $P(6>Y>3) = 0.1263$ (you can calculate this in R via 1 - sum(dpois(seq(0, 5), 2)) and sum(dpois(c(4, 5), 2)).

Overall, since you are sampling from a distribution that is not $\text{Poisson}(2)$, your sampling distribution does not look like it should, and since you are sampling from a distribution that has fairly marked differences from a $\text{Poisson}(2)$, your sampling distribution looks quite wrong.

The good news is that (by the Glivenko-Cantelli theorem), as you get a large sample size, your empirical distribution tends toward the true distribution, explaining why the sampling distribution looks better when you up the sample size: the sample just looks more like a $\text{Poisson}(2)$, so anything derived from it looks more like it can from a $\text{Poisson}(2)$ distribution.

For a really extreme example, suppose you sampled from $N(0,1)$ and got only positive values, by "bad luck" as you wrote (which is really just ordinary luck, as that can happen every so often). You know that the sampling distribution of the usual mean $\bar X$ in this case is $N(0, \frac{1}{\sqrt{n}})$. However, when you run the bootstrap, you will only select positive values and will only calculate positive means, meaning that the bootstrap sampling distribution gives no probability to values below zero, despite the fact that $N(0, \frac{1}{\sqrt{n}})$ has half of its density allocated to negative numbers.

$\endgroup$
2
  • 5
    $\begingroup$ +1. But note that a qq plot of the data against the Poisson distribution with the same mean is remarkably linear, indicating there really isn't anything out of the ordinary in this sample. $\endgroup$
    – whuber
    Commented Nov 22, 2023 at 20:07
  • 3
    $\begingroup$ +1, and really, a q-q-plot is overkill: just simulate drawing $n=11$ realizations from a Poisson distribution with $\lambda=2$, do this 20 times, and plot the 20 histograms in a $5\times 4$ grid. You immediately see that they all look similarly "un-Poisson-y" as the one here. The histogram in the question is absolutely run-of-the-mill for this sample size. $\endgroup$ Commented Nov 24, 2023 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.