Are bootstrapped samples considered to be coming from the same distribution as the original sample?

Question

Let a dataset $\mathcal{D}$ be sampled according to $F_{\mathcal{D}}$.

My question is, suppose I create bootstrapped samples from $\mathcal{D}$. That is, create $\mathcal{D}_1, \ldots, \mathcal{D}_M$ using sampling with replacement.

Can I consider $\mathcal{D}_1, \ldots, \mathcal{D}_M$ as coming from the same distribution responsible for generating $\mathcal{D}$?

It seems to be the case, but sampling with replacement seems to create fundamentally different datasets than the original one $\mathcal{D}$. This seems to suggest that $\mathcal{D}_1, \ldots, \mathcal{D}_M$ do not follow the same distribution.

Answer 1

Are bootstrapped samples considered to be coming from the same distribution as the original sample?

NOT QUITE

In the simulation below, I draw 50 observations from a standard normal distribution.

set.seed(2023)
N <- 50
x <- rnorm(N, 0, 1)
min(x)              # -1.875067
pnorm(-1.96)        # 0.0249979

In a standard normal distribution, the probability of drawing a point below $-1.96$ is a bit more than $2\%$. When you draw bootstrap samples from the empirical distribution of x, the smallest value that can be drawn is $-1.875067 > -1.96$, so the probability of doing so is ZERO.

Since $P_{N(0, 1)}\left(X \le -1.96\right) = 0.0249979$ and $P_{\text{Boot}}\left(X \le -1.96\right) = 0$, the distributions cannot be the same.

The gist of bootstrap methods is, however, that if we can't go back and sample from the original distribution, a representative empirical distribution is the next-best option, and nice properties follow from this line of reasoning.

EDIT

From the comments, this question concerns slide ten here. In those slides, the claim is that each individual draw from the empirical distribution has the same distribution as the empirical distribution, which is true. However, the OP seems to mix up the notation for the empirical distribution and original distribution.

The $\mathcal{D}_k$ do indeed have the same distribution as the entirety of the data $\mathcal{D}$, which is just the counts of how many of each value are in the data. Since each sample samples with replacement, the probability of getting any individual value for $\mathcal{D}_k$ is whatever the proportion of $\mathcal{D}$ that value represents. However, the $\mathcal{D}_k$ do not have the same distribution as that which generated $\mathcal{D}$ unless we are lucky enough to have a perfect representation of the original data (such as flipping $50$ heads and $50$ tails for a fair coin).