If all you are doing is re-sampling from the empirical distribution, why not just study the empirical distribution? For example instead of studying the variability by repeated sampling, why not just quantify the variability from the empirical distribution?

  • 8
    $\begingroup$ "(In this sense,) the bootstrap distribution represents an (approximate) nonparametric, noninformative posterior distribution for our parameter. But this bootstrap distribution is obtained painlessly - without having to formally specify a prior and without having to sample from the posterior distribution. Hence we might think of the bootstrap distribution as a “poor man’s” Bayes posterior." Hastie et al. The Elements of Statistical Learning". Sect. 8.4. $\endgroup$ – usεr11852 Mar 4 '18 at 15:36
  • 8
    $\begingroup$ How would we quantify the uncertainty of our estimates from the empirical distribution? $\endgroup$ – usεr11852 Mar 4 '18 at 15:39
  • 2
    $\begingroup$ "Under mild regularity conditions, the bootstrap yields an approximation to the distribution of an estimator or test statistic that is at least as accurate as the approximation obtained from first-order asymptotic theory ". unc.edu/~saraswat/teaching/econ870/fall11/JH_01.pdf. $\endgroup$ – jbowman Mar 4 '18 at 16:35
  • 10
    $\begingroup$ You are arguing, not trying to understand. Believe me, you have not come to a realization that the bootstrap is worthless contra to that of many thousands of statisticians over four or so decades. You did not read the quote carefully. I think you have failed to grasp the key role randomness plays in statistics. Statements like "Why bother!!" with respect to "get a distribution of $T(X)$ are... unusual, to say the least. If you don't think it important to understand the distribution of your estimates, you might want to consider why the field of statistics exists at all, and re-think that. $\endgroup$ – jbowman Mar 4 '18 at 16:42
  • 5
    $\begingroup$ @ztyh You say "if you map each sample $X$ to $T(X)$ you get a distribution of $T(X)$". Perhaps you should think about this, how would you map a single point $X_i$ to $T(X) = \bar{X}$? Or any function $T(X_1, X_2, \cdots X_n)$ for that matter. $\endgroup$ – knrumsey Mar 4 '18 at 20:07

Bootstrapping (or other resampling) is an experimental method to estimate the distribution of a statistic.

It is a very straightforward and easy method (it just means you compute with many random variants of the sample data in order to obtain, an estimate of, the desired distribution of the statistic).

You most likely use it when the 'theoretical/analytical' expression is too difficult to obtain/calculate (or like aksakal says sometimes they are unknown).

  • Example 1: If you do a pca analysis and wish to compare the results with 'estimates of the deviation of the eigenvalues' given the hypothesis that there is no correlation in the variables.

You could, scramble the data many times and re-computing the pca eigenvalues such that you get a distribution (based on random tests with the sample data) for the eigenvalues.

Note that the current practices are gazing at a scree plot and apply rules of thumb in order to 'decide' whether a certain eigenvalue is significant/important or not.

  • Example 2: You did a non-linear regression y ~ f(x) providing you with some estimate of bunch of parameters for the function f. Now you wish to know the standard error for those parameters.

Some simple look at the residuals and linear algebra, like in OLS, is not possible here. However, an easy way is to compute the same regression many times with the residuals/errors re-scrambled in order to get an idea how the parameters would vary (given the distribution for the error term can be modeled by the observed residuals).

  • 2
    $\begingroup$ I think your example is not a bootstrap. Its just sampling from a known null distribution. Bootstrap is where you have one sample and repeatedly sample again from that sample. $\endgroup$ – ztyh Mar 4 '18 at 16:55
  • 4
    $\begingroup$ In your question you imagine to calculate the variance of a sample, which is indeed simple and does not require bootstrapping. In my example I speak about a situation in which we have a value which is derived from the sample. Then we can not simply compute a variance anymore, still we wish to know how it varies. By scrambling the data many times and re-computing the pca eigenvalues you can get such a distribution (random) data that follows the distribution of your sample. If I am not mistaken this is called bootstrapping. $\endgroup$ – Sextus Empiricus Mar 4 '18 at 17:01
  • 1
    $\begingroup$ Ok, I see where I was misunderstanding things. Your example makes sense. Thanks. $\endgroup$ – ztyh Mar 4 '18 at 17:07

The key thing is that the bootstrap isn't really about figuring out features of the distribution of the data, but rather figuring out features of an estimator applied to the data.

Something like empirical distribution function will tell you a fairly good estimate of the CDF from which the data came from...but by in isolating, it tells you essentially nothing about how reliable the estimators we build from that data will be. This is the question answered by using bootstrap.

  • 1
    $\begingroup$ Using the (non-parametric) bootstrap to find "the distribution of the data" would be a laugh: it merely comes up with the empirical distribution function, which is exactly the set of data the analyst began with. Reminds me of college algebra when I'd "solve for X" and find "X=X". $\endgroup$ – AdamO Mar 16 '18 at 18:49

IF you know exactly what is the underlying distribution, then you don't need to study it. Sometimes, in natural sciences you know exactly the distribution.

IF you know the type of the distribution, then you only need to estimate its parameters, and study it in the sense you meant. For instance, sometime you know a priori that the underlying distribution is normal. In some cases you even know what is its mean. So, for normal the only thing that is left to find out is the standard deviation. You get the sample standard deviation from the sample, and voila, you get the distribution to study.

IF you don't know what is the distribution, but think that it is one of the several in the list, then you could try to fit those distribution to data, and pick the one that fits best. THEN you study that distribution.

FINALLY, often you don't know type of distribution you're dealing with. And you do't have a reason to believe that it belongs to one of 20 distributions that R can fit your data to. What are you going to do? Ok, you look at mean and standard deviations, nice. But what if it's very skewed? What if its kurtosis is very large? and so on. You really need to know all the moments of distribution to know, and study it. So, in this case non parametric bootstrapping comes handy. You don't assume much, and simple sample from it, then study its moments and other properties.

Though non-parametric bootstrapping is not a magical tool, it has issues. For instance, it can be biased. I think parametric bootstrapping is unbiased

  • 1
    $\begingroup$ I think even if you did not know the true distribution, many moments are easy to calculate. So I think the problem is not with not knowing the type of distribution you are dealing with. Rather it is about what kind of statistic you are trying to study. Some statistic might be hard to calculate and only then is bootstrap useful. $\endgroup$ – ztyh Mar 4 '18 at 15:57
  • $\begingroup$ Like in the comment to the question to usεr11852, actually I have doubts about benefits in regards to computability of statistics as well... $\endgroup$ – ztyh Mar 4 '18 at 16:10
  • $\begingroup$ Actually I think its still a no brainer. You map each sample to $\ln (x^3+x)$. Then finding the quantile is again a 1 line code. So 2 lines of code in total. $\endgroup$ – ztyh Mar 4 '18 at 16:22
  • 1
    $\begingroup$ quantile was a stupid example, I'll give you that. try mean instead. in my practice I have to forecast $x*z$ or even more complex functions $f(x,z)$ where $x,z$ are from an unknown joint distribution. I need to obtain properties of the final forecast. try that with moments. with bootstrapping it's a no brainer. $\endgroup$ – Aksakal Mar 4 '18 at 16:28
  • 1
    $\begingroup$ How ever complicated $f$ maybe, all you have to do is map the samples of $x$ and $z$ to $f(x,z)$. Then study those mapped samples. If you can use bootstrap, then that means you can do this, and this is much easier... $\endgroup$ – ztyh Mar 4 '18 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.