# Bootstrap vs Monte Carlo, error estimation

I'm reading the article Error propagation by the Monte Carlo method in geochemical calculations, Anderson (1976) and there's something I don't quite understand.

Consider some measured data $\{A\pm\sigma_A, B\pm\sigma_B, C\pm\sigma_C\}$ and a program that processes it and returns a given value. In the article, this program is used to first obtain the best value using the means of the data (ie: $\{A, B, C\}$).

The author then uses a Monte Carlo method to assign an uncertainty to this best value, by varying the input parameters within their uncertainty limits (given by a Gaussian distribution with means $\{A, B, C\}$ and standard deviations $\{\sigma_A, \sigma_B, \sigma_C\}$) before feeding them to the program. This is illustrated in the figure below:

where the uncertainty can be obtained from the final $Z$ distribution.

What would happen if, instead of this Monte Carlo method, I applied a bootstrap method? Something like this:

This is: instead of varying the data within their uncertainties before feeding it to the program, I sample with replacement from them.

What are the differences between these two methods in this case? What caveats should I be aware of before applying any of them?

I'm aware of this question Bootstrap, Monte Carlo, but it doesn't quite solve my doubt since, in this case, the data contains assigned uncertainties.

• Just to clarify: the "random change" in the MC method is randomly generated by the researcher? That is, noise/errors are being artificially added to the input data? – shadowtalker Sep 7 '16 at 21:08
• It is "randomly generated", based on the uncertainties of the measured data (ie: the $\sigma$s) and assuming a certain distribution for these errors (usually Gaussian). So no, errors are not artificially added. The input data has an associated error given by the measuring process. – Gabriel Sep 7 '16 at 21:11
• I don't think I understand. That is artificial noise, but with a standard deviation estimated from the data – shadowtalker Sep 7 '16 at 22:19
• Then I probably don't understand what "artificial noise" is (and what would constitute "non-artificial noise"). Have you seen the article? It certainly explains things a lot better than me. – Gabriel Sep 8 '16 at 1:29
• Natural noise: random variation in my data. Artificial noise: using a random number generator to draw numbers from a probability distribution, and adding those numbers to my data – shadowtalker Sep 8 '16 at 1:46

As far as I understand your question, the difference between the "Monte Carlo" approach and the bootstrap approach is essentially the difference between parametric and non-parametric statistics.

In the parametric framework, one knows exactly how the data $x_1,\ldots,x_N$ is generated, that is, given the parameters of the model ($A$, $\sigma_A$, &tc. in your description), you can produce new realisations of such datasets, and from them new realisations of your statistical procedure (or "output"). It is thus possible to describe entirely and exactly the probability distribution of the output $Z$, either by mathematical derivations or by a Monte Carlo experiment returning a sample of arbitrary size from this distribution.

In the non-parametric framework, one does not wish to make such assumptions on the data and thus uses the data and only the data to estimate its distribution, $F$. The bootstrap is such an approach in that the unknown distribution is estimated by the empirical distribution $\hat F$ made by setting a probability weight of $1/n$ on each point of the sample (in the simplest case when the data is iid). Using this empirical distribution $\hat F$ as a replacement for the true distribution $F$, one can derive by Monte Carlo simulations the estimated distribution of the output $Z$.

Thus, the main difference between both approaches is whether or not one makes this parametric assumption about the distribution of the data.

• Almost two years later, I know believe this to be the best answer because it explicitly mentions the difference between the parametric and non-parametric approaches (which I didn't know back then) Thus, I'm changing the accepted answer to this one. – Gabriel Jun 20 '18 at 19:11
• but for the paramrtric approach one can also use parametric bootstrap right? – Tom Wenseleers Jun 5 at 19:15

The Random Change in your Monte Carlo Model is represented by a bell curve and the computation probably assumes normally distributed "error" or "Change". At least, your computer needs some assumption about the distribution from which to draw the "change". Bootstrapping does not necessarily make such assumptions. It takes observations as observations and if their error is asymetrically distributed, then it goes into the modell that way.

Bootstrapping draws from the observation and thus needs a number of true observations. If you read in a book, that C averages at 5 with a standard deviation of 1, than you can set up a Monte Carlo Modell even if you don't have observations to draw from. If your observation is scarce (think: astronomy) you may set up a Monte Carlo Modell with 6 observations and some assumptions about their distribution but you will not bootstrap from 6 observations.

Mixed modells with some input drawn from observed data and some from simulated (say hypothetical) data are possible.

Edit: In the following discussion in the comments, the original poster found the following helpfull:

The "original program" does not care, whether it gets a value, that you computed from a mean and a deviation or that is a true realisation of a mean and a deviation in a natural process.

• Thank you for your answer Bernhard! A few questions that pop to my mind. 1. Am I understanding correctly that the only (main?) difference between these two methods is that MC needs to assume a distribution for the uncertainties while the bootstrap does not? 2. If I had a large enough dataset and I performed the iteration numerous times ($N\to\infty$), would these two methods converge then on the estimated uncertainty assigned to the best value? 3. Am I not discarding valuable data by not using the uncertainties assigned to the input data in the bootstrap method? – Gabriel Sep 7 '16 at 14:36
• I am statistically/machine-learningly self-taught, so I will not claim that any of the differences I mentioned are the only ones. I am not even certain, whether Bootstrapping is considered a Monte Carlo method itself. Both of the algorithms simulate a large number of realistic scenarios. You can either draw the input from assumptions or from observations. My field is medicine and assumptions are notoriously wrong in that field. Therefore I would try to go with observations whenever they are available in large enough numbers. It may well be, that in field closer to physics or chemistry,... – Bernhard Sep 7 '16 at 14:46
• ...that in fields closer to physics or chemistry, assumptions are more reliable. As to point 2: If you go by large enough samples and iterations I assume, you would find that real data is never truely normally distributed and that your assumptions are always a little bit wrong, but I can't claim any knowledge. As to Point 3: I am not shure to have understood what you mean by discarding valuable data in the bootstrap method. "Assigning uncertainty" is man-made, Data comes from reality. Again, this is my belief based upon my field. In reality, you will rarely have good theory and large data – Bernhard Sep 7 '16 at 14:52
• By discarding valuable data I mean that the bootstrap method makes no use of the uncertainties assigned to the data (ie: $\sigma_A, \sigma_B, \sigma_C$) This is "information" that the MC method takes into account but the bootstrap discards. – Gabriel Sep 7 '16 at 15:15
• Each observation is a measured value and thus already contains it's own measurement error and uncertainty. The "original program" does not care, whether it gets a value, that you computed from a mean and a deviation or that is a true realisation of a mean and a deviation in a natural process. But of course, all resampling techniques rely on a large data basis and you can compute arbitrary numbers or random numbers but usually not make arbitrary numbers of observations. So in cases where you have a large number of observations, I don't see, where data is discarded. – Bernhard Sep 7 '16 at 15:22

If the function relating the output Z to the inputs is reasonably linear (i.e. within the variation range of the inputs), the variance of Z is a combination of the variances and covariances of the inputs. The details of the distribution do not matter too much... So, both methods should return similar results.

See the Supplement 1 to the GUM

• What happens when the function is not reasonably linear? How will these two methods differ then? – Gabriel Sep 7 '16 at 14:52
• In that case, you should refer to the answer above, by Bernhard. That is, for them to coincide, you should have a faithful description of the data pdf for Monte Carlo. – Pascal Sep 9 '16 at 9:51

Bootstrap means letting the data speak for themselves. With Monte Carlo method, you sample many random draws from the imposed CDF (normal; gamma; beta...) via uniform distribution and create an empirical PDF (provided that the CDF is continuous and derivable). An interesting explanation of the whole Monte Carlo process is reported in: Briggs A, Schulper M, Claxton K. Decision modelling for health economic evaluation. Oxford: Oxford University Press, 2006: 93-95.