Difference in Distributions: Monte Carlo simulation vs Reality

Question

We are doing fire probability modeling using a Monte Carlo approach. Essentially, we use historical fire statistics and weather culled from a 50 year period to develop inputs to a fire model that then draws from these distributions to burn fires for a single season. We then run ~50,000-100,000 simulations and end up with a whole bunch of fires designed to reveal burn probability.

Many users of these types of models insist that the outputs of the Monte Carlo simulations should be "realistic", and that a curve of the frequency distribution of fires sizes from the model output (that is a frequency distribution the sizes of ~10's of thousands of iterations) should have the same relative distribution of fire sizes as the 50 years of "real" data that populated the model.

I am not convinced that this is necessary, because "reality" in this case is just 50 "iterations" as opposed to the 10's of thousands of iterations from the Monte Carlo simulations:

How can I show that the "real" distribution is not significantly different than the "simulated" distribution? Due to the stochastic nature of the processes involved in the model, they are bound to be different to some degree. How much difference matters?

Answer 1

This is called posterior predictive checks and there is no single solution for doing them. Obviously simulated data won't be the same as real data. Even if you use procedures that stay close to the empirical distribution like bootstrap, or sampling from kernel densities, in the end you receive smoothed approximation of the original distribution. The whole idea of statistical modeling is that we wan to describe complicated real-life events in simpler form, to reach some general conclusions. Real data will always be more noisy then the simulated data. The whole point is to conduct your simulation in such manner that makes the simulated data close enough to the real data. "Close enough" is arbitrary and depends on your aims. More precisely, on what properties of the real data you are interested in. So to answer yourself the question, you need to decide what would make you (and possibly your audience, or customers) consider the simulated data to be close enough.