Let's say we have a dataset $D$ with $N$ rows and $M$ columns. Each column is a feature. And for each feature $X_1, X_2,..., X_N $~ iid $F_p$ where $F_p$ is the distribution for feature p. Now let's do a random sampling which samples $K$ rows from $D$ and let's denote this sample as $D^K$.

So my question is: in $D^K$, for each feature p, does it follow the same distribution of this feature p in $D$? My instinct is that when $M$ is a large number, if $K$ is not large enough, then it cannot guarantee every feature p in $D^K$ has the same distribution as that in $D$. But I don't know if it is right. If it is right, how can I determine the minimum $K$ that's required?


  • $\begingroup$ If you sample with replacement from the sample you're sampling from $\hat F$ rather than F. If you're randomly sampling the population the sample cdf converges to the population cdf, but resampling the sample cdf will converge to that, not to F $\endgroup$
    – Glen_b
    Jul 12, 2019 at 3:51
  • $\begingroup$ yea, I intended to say sample without replacement. $\endgroup$
    – lun yu
    Jul 29, 2019 at 17:39
  • $\begingroup$ If you sample without replacement it's even worse, you just get $\hat{F}$ every time. $\endgroup$
    – Glen_b
    Jul 29, 2019 at 22:32

1 Answer 1


If you forbid replacement, then the answer is no. If you allow replacement, you’re doing something called bootstrap. The idea of bootstrap is that, if we can’t collect more samples from the population distribution, the next-best approach is to sample the empirical distribution.

Explicitly, your $D^k$ sample will not follow the original population unless the population is discrete and $D$ perfectly recreates the discrete PMF. But with large enough sample sizes, you should get close.

  • $\begingroup$ so if sampling without replacement, the sample distribution will not be the same as the population distribution regardless of sample size? and also for boostrap, is there any requirement for sample size of each boostrap? $\endgroup$
    – lun yu
    Jul 29, 2019 at 17:48
  • $\begingroup$ @lunyu 1) Unless you've perfectly sampled the population, no. 2) The way bootstrap goes is that if you have a sample size of N, your sample with replacement is also of size N. Some observations may be repeated, and some may be omitted, yes. $\endgroup$
    – Dave
    Jul 29, 2019 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.