On my course of learning statistical methods, I have read about a boostrapping technique where to estimate some statistic $t$ on data $X$ we sample with replacement from the original data and obtain the estimates $t(X_1),\dots,t(X_n)$. For example, we can use $\bar t_n(X) := \frac1n\sum t(X_k)$ instead of $t(X)$. My question is: it seems that $\bar t_n(X) \to t(X)$ for linear statistic $t$, is that correct? Does it mean then that the bootstrapping technique is more useful to estimate some non-linear statistics?


2 Answers 2


Bootstrap lets you to approximate the true distribution of your variable $X$, so that you can compute different statistics on bootstrap samples that approximate the true values. Here you can read an extended definition and introduction to bootstrap: Explaining to laypeople why bootstrapping works .

The general idea is that you sample with replacement $N$ cases out of $N$ cases in the dataset. You repeat this a number of times, say $R$, and so those bootstrap samples let you approximate the true, unknown, distribution of your variable. You can estimate any statistic on those samples, it does not have to be linear. Bootstrap lets you approximate the distribution of $X$, so you can use it with any statistic that can be estimated using your $X$ variable.

The power of this method is that it is simple and general and so can be used for a number of different methods. Bootstrap is also helpful because it can be used when there is no direct analytical solution, e.g. you want to estimate confidence intervals for data that comes from skewed distribution (e.g. here). On another hand, there are bootstrap solutions for problems that have analytical solutions, so they can be used as an alternative method (e.g. here). For more examples of using bootstrap check the book by Davidson and Hinkley "Bootstrap Methods and Their Applications" (more here).

  • $\begingroup$ Thanks, I'm not confusing the samples with observations: to be precise in notation let's say we start with observations $X = (x_j)_{j=1}^m$, and let $p_j$ be a frequency of $x_j$ in $X$ so that $\sum p_j = 1$. This determines empirical distribution of $X$; we sample from it $X_1,\dots,X_n$ so that $\Bbb P(X_{ij} = x_j) = p_j$. So my question is why do we use samples from empirical distribution if we have a direct access to it? $\endgroup$
    – Ulysses
    Jan 2, 2015 at 9:14
  • $\begingroup$ I have a feeling that you are confused: in my comment the data is denoted by small letters $x_j$ which range to $m$, and the samples are denoted with capital letters and range to $n$. E.g. taking your example one of the samples can be $X_{11} = x_1$, $X_{12} = x_1$, $X_{13} = x_7$, $X_{14} = x_9$, $X_{15} = x_5$ etc. $\endgroup$
    – Ulysses
    Jan 2, 2015 at 10:43
  • $\begingroup$ For sure I have read the linked answers, and I've asked my question in the comment since I did not find them to answer my question. Again, we have a direct access to the empirical distribution so if we want, we can find all distribution of statistics we study in bootstrapping analytically. So to me it only seems that bootstrapping is just a short name for Monte Carlo method that samples from the empirical distribution, since it can be possible but hard to get these analytical formulas (if the data is big). Is that correct? $\endgroup$
    – Ulysses
    Jan 2, 2015 at 10:46
  • $\begingroup$ Yes, bootstrap is helpful if there is no analytical solution, but not only (see my edited answer). And about Monte Carlo, it depends of the definition of "Monte Carlo" you use. $\endgroup$
    – Tim
    Jan 2, 2015 at 18:32

Bootstrapping enables to approximate the "true" distribution of the statistic t (wikipedia).

By construction $\lim \limits_{n \to \infty} \bar{t}_n(X) = t(X)$. However, as said above, bootstrapping is normally not used for estimation of the statistic itself but for its distribution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.