This is a follow-up on the terminology used in this answer. In brief, I was surprised by the term bootstrapping being used more broadly than what I have mostly encountered so far.

My Understanding of the Terminology so far

The following is is based on several articles and other material on bootstrapping. Admittedly they were all rather vague, so I am attempting a definition below. A statistics professor confirmed the main distinction I am making here.


Given a real dataset $X$ sampled from some population $P$, a bootstrap is an artificial dataset that aims to represent another sample from the population $P$, but is constructed using only knowledge of $X$. Bootstrapping is any technique exploiting the statistical properties of bootstraps.

For example if we have a dataset $X$ of paired numbers, we can generate a bootstrap by resampling this dataset, leaving the pairs intact. Let $r$ being a correlation coefficient and $\hat{R}$ the distribution of $r$ of an appropriate amount of such bootstraps. We can then estimate the confidence interval of $r(X)$ by looking at the width of $\hat{R}$. However, we would expect $r(X)$ to be somewhat central within $\hat{R}$ (i.e., not an outlier).

Not Bootstrapping

I have seen this being called Monte Carlo null model, just null model, or surrogate. I here use the latter, because it is more compact, but I acknowledge that it is not widely used.

A surrogate is a dataset that aims to represent a null model but inherits some properties from the original dataset $X$. We can build a hypothesis test based on this by comparing a statistic for $X$ and the surrogates, where we reject the corresponding null hypothesis when the statistic for our original dataset is more extreme than for the surrogates.

In the correlation example, suppose our null hypothesis is that the data is uncorrelated. We can produce a corresponding surrogate by repairing our dataset (thus destroying the pairs). Let $\tilde{R}$ be the distribution of $r$ of an appropriate amount of such surrogates. If our null hypothesis is false, we expect $r(X)$ to be extreme in comparison to $\tilde{R}$, i.e., be an outlier. Otherwise, we expect $r(X)$ to lie within $\tilde{R}$.

The Simulations in Question

My understanding of the simulation part of this answer by W. Huber is this: Our null model is that the data is generated by an inhomogeneous Poisson process. We estimate the event rate of this process from the original data via a GLM. Then we simulate 2000 artificial datasets complying with the null model, i.e., instances of the inhomogeneous Poisson process. As the dispersion for the original data is much lower than for all artificial datasets, we can reject the null hypothesis: The dispersion of the original data is significantly low.

I was surprised by the usage of bootstrapping for this: The artificial datasets here do not aim to reproduce whatever population generated the original data. We do not try to estimate an interval of confidence for the dispersion of the underlying population or similar. I would call the artificial datasets surrogates. I remarked this to which W. Huber replied:

Yes, it is honest-to-God bootstrapping. There are various flavors. This one is parametric in the sense of assuming the data arise as independent realizations of Poisson variables--in effect, an inhomogeneous Poisson process. There is no "null model" or other hypothesis in effect.

Actual Question

  • Is there any somewhat authoritative resource that confirms or refutes my definitions given above?
  • In case of confirmation, did I somehow miscategorise W. Huber’s approach?
  • In case of refusal, is there any terminology to distinguish the two kinds of artificial datasets (what I call bootstraps and surrogates above).
  • Either way, how is there not a null model and hypothesis as I identify above?
  • 2
    $\begingroup$ Your misunderstanding is expressed in "The artificial datasets here do not aim to reproduce whatever population generated the original data." On the contrary, they do--but they do so by adopting a model for that data-generation process. (This is what makes it a parametric bootstrap, or at least semi-parametric). Because the bootstrapped samples consistently differ in an important way from the original data, we conclude the model is incorrect. $\endgroup$
    – whuber
    Commented May 22, 2020 at 17:02
  • $\begingroup$ @whuber: While I see your point, that’s not exactly what I meant by aim. For example, in classical bootstrapping by resampling, you only assume that your data is representative of the underlying population and do not question this (as you cannot know better). In what you do, whether your bootstraps actually represent the population is your question. $\endgroup$
    – Wrzlprmft
    Commented May 24, 2020 at 9:13
  • $\begingroup$ Thank you, that's a useful distinction. $\endgroup$
    – whuber
    Commented May 24, 2020 at 16:33

2 Answers 2


What you call "non bootstrapping" is actually a parametric bootstrapping. That's what @whuber is doing in the answer/post that prompted your question.

Here's a semi-formal definition of a bootstrap from Babu, G. J., and Rao, C. R. (2004). Goodness-of-fit tests when parameters are estimated. Sankhya, 66, no. 1, 63-74.:

enter image description here

The idea's that you estimate the parameters of the distribution, then generate sample from the distribution with estimated parameters. Now you can study sampling distribution of parameters. That's what you called "Monte Carlo null."

This presentation explains the method in good detail. The application was Kolmogorov-Smirnov goodness-of-fit test. It's a very popular GoF for distribution fitting. The problem is that one needs to know the true distribution, i.e. its parameters are not estimated from data. When parameters are estimated, then the test statistics can be generated by parametric bootstrapping. The author of the presentation has detailed papers on the subject, they're quite well written. For instance, KS test application is explained in this one: Babu, G. J., and Rao, C. R. (2004). Goodness-of-fit tests when parameters are estimated. Sankhy¯a, 66, no. 1, 63-74.

Here's another paper on parametric bootstrapping used in connections with Bayesian inference to generate the posterior distributions: Efron B. Bayesian inference and the parametric bootstrap. Ann Appl Stat. 2012;6(4):1971‐1997. doi:10.1214/12-AOAS571 url: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3703677/

It's just another application of the same idea, which you describe as "Monte Carlo null." There's any number of papers on this subject, you may pick any that you consider authoritative.

  • $\begingroup$ +1 That is indeed an interesting slide set. $\endgroup$
    – whuber
    Commented May 23, 2020 at 16:41
  • $\begingroup$ Thank you for your answer. However, going through the linked slides, the definition of bootstrapping and the subsequent examples for bootstrapping seem to align with the characterisations with bootstrapping I gave in my question, but then I did not go through everything, as it is rather long and I did not have the background knowledge for some parts. The paper on the other hand does not seem to define bootstrapping altogether. Can you quote a relevant statement? $\endgroup$
    – Wrzlprmft
    Commented May 24, 2020 at 8:56
  • $\begingroup$ On another note, can you comment the issue of distinguishing what I call “not bootstrapping”? As far as I understand, parametric bootstrapping in your sense is at best superset of this as it also covers regular bootstrapping applications (as opposed to hypothesis tests). $\endgroup$
    – Wrzlprmft
    Commented May 24, 2020 at 8:59

For whatever it’s worth, I stumbled upon a publication that is essentially about this, namely namely Jason H. Moore, Phys. Med. Biol. 44, L11 (1999) (letter to the editor). The author notes that the method of surrogates is a form of permutation testing and distinct from bootstrapping:

[…] bootstrapping […] is based on resampling data with replacement with the goal of creating an empirical sampling distribution of the statistic. In essence, bootstrapping treats the sample as the population and is used when the population distribution of the statistic is unknown and thus supports inferences about population parameters.

Applying their categorisation the simulation in question is clearly not bootstrapping, but some sort of test (albeit not necessarily a permutation test).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.