3
$\begingroup$

Let a dataset $\mathcal{D}$ be sampled according to $F_{\mathcal{D}}$.

My question is, suppose I create bootstrapped samples from $\mathcal{D}$. That is, create $\mathcal{D}_1, \ldots, \mathcal{D}_M$ using sampling with replacement.

Can I consider $\mathcal{D}_1, \ldots, \mathcal{D}_M$ as coming from the same distribution responsible for generating $\mathcal{D}$?

It seems to be the case, but sampling with replacement seems to create fundamentally different datasets than the original one $\mathcal{D}$. This seems to suggest that $\mathcal{D}_1, \ldots, \mathcal{D}_M$ do not follow the same distribution.

$\endgroup$

1 Answer 1

3
$\begingroup$

Are bootstrapped samples considered to be coming from the same distribution as the original sample?

NOT QUITE

In the simulation below, I draw 50 observations from a standard normal distribution.

set.seed(2023)
N <- 50
x <- rnorm(N, 0, 1)
min(x)              # -1.875067
pnorm(-1.96)        # 0.0249979

In a standard normal distribution, the probability of drawing a point below $-1.96$ is a bit more than $2\%$. When you draw bootstrap samples from the empirical distribution of x, the smallest value that can be drawn is $-1.875067 > -1.96$, so the probability of doing so is ZERO.

Since $P_{N(0, 1)}\left(X \le -1.96\right) = 0.0249979$ and $P_{\text{Boot}}\left(X \le -1.96\right) = 0$, the distributions cannot be the same.

The gist of bootstrap methods is, however, that if we can't go back and sample from the original distribution, a representative empirical distribution is the next-best option, and nice properties follow from this line of reasoning.

EDIT

From the comments, this question concerns slide ten here. In those slides, the claim is that each individual draw from the empirical distribution has the same distribution as the empirical distribution, which is true. However, the OP seems to mix up the notation for the empirical distribution and original distribution.

The $\mathcal{D}_k$ do indeed have the same distribution as the entirety of the data $\mathcal{D}$, which is just the counts of how many of each value are in the data. Since each sample samples with replacement, the probability of getting any individual value for $\mathcal{D}_k$ is whatever the proportion of $\mathcal{D}$ that value represents. However, the $\mathcal{D}_k$ do not have the same distribution as that which generated $\mathcal{D}$ unless we are lucky enough to have a perfect representation of the original data (such as flipping $50$ heads and $50$ tails for a fair coin).

$\endgroup$
5
  • $\begingroup$ Thank you Dave. This makes me more confused actually. Became I am looking at this in the context of bias and variance trade off. See page 10/23 by a Professor at Cornell which says the bootstrapped samples D_k has the same distribution as the original one. people.orie.cornell.edu/mru8/orie4741/lectures/… $\endgroup$ Nov 16, 2023 at 16:34
  • $\begingroup$ @ShamisenExpert Isn't that $\mathcal{D}$ on slide ten the empirical distribution and not the original distribution? I think that $\mathcal{D}$ is what I call x. $\endgroup$
    – Dave
    Nov 16, 2023 at 16:35
  • $\begingroup$ Finite samples is, impo, nearly the opposite of “in the limit of many samples”. 50 < 20k < infinity $\endgroup$ Nov 16, 2023 at 16:59
  • $\begingroup$ @EngrStudent Who said anything about "in the limit of many samples"? $\endgroup$
    – Dave
    Nov 16, 2023 at 17:01
  • 1
    $\begingroup$ I said it. :) In general, for continuous distributions, the “true” distribution has infinite number of values in the domain. It is rare to get a summary statistic on sampled data that matches the analytic result to round off. More rare two get two fundamentally different statistics matching to round off. Many samples reduce the error but not all the way to zero. Mean is fun but 12th centered moment is harder. $\endgroup$ Nov 16, 2023 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.