Monte-carlo simulation and extrapolation

Question

I am reviewing some work and the proposed solution seems to me not to be reliable. But I fail to find any references or even consistently formulate why I think this approach does not work.

Assume you want to model the risk of two objects colliding in a given space and the probability of a collision is expected to be small, say somewhere in the order of $10^{-6}$. A collision event is defined as the event when the radii of the two objects overlap in space. To model this, the approach runs about $10^5$ simulations and stores the minimum distance between the two objects for each run (i.e., the distance of the closest point the two objects came to on another in each run). Once the simulations are done, a number of probability functions are fitted to the minimum distance and the best function is kept (based on AIC). So far, that approach seems reasonable (except maybe for the choice of AIC as a useful measure of the goodness of fit). Now the proposed approach uses the fitted probability density function to extrapolate to the area where the distances of the two objects is so small that they collide. I tried to sketch the histogram in the figure below. Note that it’s not to scale and I haven’t even seen the data, so the actual distribution of doesn’t really matter. I am interested in knowing if this approach works in general, not in a specific case.

Intuitively, this approach does not make sense to me. It assumes that the probability distribution which fits the data also fits on the far ends of the tails (where we wouldn't expect many, if any, observations from the simulations). Also, it seems counterintuitive that you can have a reliable estimate of the probability of an event in the order of $10^{-6}$ when you run only $10^5$ simulation runs.

Has anyone a more stringent argument why this shouldn’t (or perhaps should) work and/or can point me to some references?

Answer 1

There are indeed approaches which are better suited to estimate such extreme outcomes. The key word is Extreme Value Theory. A standard reference is Modeling extremal events by Embrechts, Klüppelberg and Mikosch.

The problem with the approach is exactly as you describe: Why can one extrapolate into regions where one has not seen any data (yet)? Fitting "some" parametric model on the body and then hoping that this describes the tails may or may not work in practice. But extreme value theory provides reasons why extreme quantiles can be modelled with certain quite specific parametric distributions.

Just judging from what you wrote a "Peaks over threshold" approach may be a good starting point. In this method you fit an extreme value distribution called Generalised Pareto distribution (whose parametric form can be justified by mathematical arguments) to the "interesting" tail of your simulated data.

Have a look at the reference where all this is explained in detail. For a more applied/readable treatment have a look at the book Quantitative Risk Management Chapter 7.2 "Threshold Exceedances" and there specifically Section 7.2.4 The Hill Method

Answer 2

Extrapolation is not bad

If you could model this based on a mechanistic/ab-initio foundation, but you are just lacking some experimental parameters or some stuff that needs to be computed/estimated with a Monte Carlo simulation, then this could work well.

Extrapolation is not bad if you have good reasons to know that your model is correct and if you consequently know that variables estimated in the range that you are measured are gonna be good as well outside the range.

Extrapolation is bad

However, if you are just using some fitting of uninspired models without a theoretical foundation and are guessing based on best AIC which model could be working well, then the extrapolation is having no theoretical foundation at all and may likely be biased.

---

So, you need some underlying theory to justify the extrapolation.

That theory might be wrong, but given that the theory is correct, or assuming that the theory is approximately close to the reality, you can apply the extrapolation. In this case it is at least clear how one gets to the final conclusions and what the underlying assumptions are.

Without a theory, and just extrapolating data according to certain curves about which we do not know why they should be general except for the fact that they happen to fit the small range well, there is nothing that indicates that the conclusions are trustworthy outside the observed range.