0
$\begingroup$

[Edit: I include a specific example for clarity]

Say I have two models:

Model 1) fuel_consumption_model, which calculates the fuel consumption of barges. It has about a dozen input parameters that all have distributions. It takes about 10 minutes to run 1000 times.

Model 2) emissions_model, which calculates the emissions of barges based on several independent parameters, one of which is fuel consumption

I want to run Model 2 1000 times to get the distribution of emissions (the model output). I could link up the two models, but that would add a lot of computing time because I would be recalculating the fuel_consumption_model outputs.

So instead I:

(1) run Model 1 N times, and store the results (fuel consumptions) in a Nx1 array;

(2) reuse the fuel consumption array directly in Model 2. Specifically, I randomly take a value from the Model 1 output array for each Monte Carlo Simulation iteration.

I'd like to read up on the use of this approach in other fields, and also read the opinion of the statistical community on this type of approach. Unfortunately, my search has not yielded anything.

I've read this about preemptively sampling a known distribution, and I understand the criticism, but in my case, it would be unthinkable to generate samples for my input parameters every time the large model is run, nor to describe parametrically the distribution of these input parameters.

I've also read on non-parametric models, but am not sure it entirely applies to what I am doing.

I'd appreciate any pointers, key words or opinions.

$\endgroup$
  • $\begingroup$ Why do you write $N\times 1$? Aren't there many input parameters (say $d$) and the $N$ sets would need a $N\times d$ array? And what are you resampling exactly? Once you know the output for each of the $N$ sets you are done and nothing could be gained from sampling from those $N$ again because the results for each sample will be known already??? $\endgroup$ – g g Jun 20 at 21:40
  • $\begingroup$ @gg (1) Indeed, if I have d parameters, then my array of sampled values is d x N (I should have written $1 x N$ above, but the transpose doesn't affect my question. $\endgroup$ – MPa Jun 21 at 15:29
  • $\begingroup$ @gg (2) I don't want to sample them for the sake of sampling them - I want to sample them to use values in another model. So, for example: I have a real complicated model for estimating fuel consumption of barges, that has many input parameters. I run the model N times, store the results, and then use these fuel consumption results as samples in another model that calculates emissions from barge transport. In this example, suppose recalculating fuel consumption within the context of the emissions model is not feasible. $\endgroup$ – MPa Jun 21 at 15:32
  • $\begingroup$ Your comment (2) confuses me even more! Before you talked about inputs and a large model. Now you have the inputs, you have the fuel consumption model for barges and then the fuel consumption model in the context of the(?) emissions model. I think you really need to clarify at least on an abstract level the inputs, the large model(s), the data flow between them, what you want to sample and why you need to sample. If you apply (a chain of) deterministic functions to a sample you are not sampling from a sample. $\endgroup$ – g g Jun 21 at 15:59
  • $\begingroup$ @gg Included an example from the get-go, hope this clarifies things. $\endgroup$ – MPa Jun 21 at 17:48
2
+50
$\begingroup$

I try to summarize your situation in more mathematical terms, since this simplifies the argument. For your first model, you feed inputs $X$, a vector of independent random variables, to the model $f$ such that the outputs are $f(X).$ Furthermore you have a second model $g$, which takes inputs $Y$ and $f(X)$ to create output $g(Y,f(X)).$

Math now tells you that for independent $X$ and $Y$ also $f(X)$ and $Y$ are independent. This means: If $(x_n)$ and $(y_n)$ are iid samples of size 1000 then $(f(x_n),y_n)$ is a perfect iid sample of size 1000 for the joint input distribution of your model $g$.

Quite specifically, there is no need (and you should absolutely NOT do it) to sample from the empirical distribution $f(x_n)$ with replacement.

To see why (re)sampling from $f(x_n)$ is a bad idea consider:

  • Some instances of the full sample $(f(x_n))$ will be missing from your resample. I.e. you will have no information how $g$ might behave if one of these instances is input.
  • Some instances of the full sample $(f(x_n))$ will appear more than once. Something which would not happen if you sample from continuous distributions and which makes such a (re)sample less realistic. In addition you will gain less information, since you evaluate your model $g$ at a position which is partially known already.

Since the best advice is to "just go ahead and use the samples" this method has no special name. Except maybe: sampling?

Finally some keywords: What you are fundamentally interested in seems to be "design of (computer) experiments" and "uncertainty quantification". Standard reference (although heavy on the "non-parametric models") is Design and Analysis of Computer Experiments

$\endgroup$
  • $\begingroup$ Thanks @gg. I'm sure to get my hands on Design and Analysis of Computer Experiments. $\endgroup$ – MPa Jun 26 at 1:43
0
$\begingroup$

I think you're correct that the issues regarding preemptively sampling known distributions don't really apply to this problem. You're simply taking the the output from a model and using it as one of many input variables to another model. As long as the fuel_consumption_model is a sufficiently "good" (however you define that) model for your purposes, there should be no problem using the fuel output in your emissions model.

For example, it'd be no different than using something like land-use classes (which are themselves the product of a model) in a model predicting property values. Whether it is "ok" depends entirely on how accurate you need the model to be.

$\endgroup$
  • $\begingroup$ Thanks, but I'd like to read up on the use of this approach in other fields. Does this type of approach (saving arrays of model outputs, randomly sampling from this array during Monte Carlo Simulation of another model where it is an input) have a name? $\endgroup$ – MPa Jun 21 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.