2
$\begingroup$

Let us say I have a model with good predictive performance. The model has a few parameters X1, X2, X3, X4 and predicts Y. I would like to perform some 'what if scenario simulations' using this model by, for example, reducing X3 by 1 and adding 2 to X4 to see what happend with Y.

I was thinking about the following approach

(1) Sample (with replacement?) from my original dataset to obtain SampleData and obtain a prediction called Old given my model.

(2) I would then apply:

SampleData$X3 <- SampleData$X3 - 1
SampleData$X4 <- SampleData$X4 + 2

and obtain a prediction called New given my model.

I would repeat (1) and (2) a few thousand times. I could then draw the distributions of Old and New and see, for example, if New is significantly greater than Old.

I guess this is bootstrapping of my model? Does this make sense? Could I get CIs for this (e.g. the difference between Old and New)?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ This makes sense for a what if scenario, and cheers for using bootstrapping to gather information on the robustness of the result. If you want to go a step further you could also fit your model on bootstrap samples to account for more variance. A word of warning though, be careful about interpreting these results causally. If you actually plan to change whatever X3 represents in real life, there is no guarantee that your model has captured the effect of an intervention like that unless the data you used to train the model also contains that intervention. $\endgroup$
    – Matthew Drury
    Commented Oct 23, 2018 at 14:23
  • $\begingroup$ Thanks. This is actually what I was thinking as well. My question is related to this one: stats.stackexchange.com/questions/372953/… Do you know how to obtain the CIs from the differences? I think there are some standard formulas but I am still not 100% sure if this is bootstrapping and if to use resampling ... $\endgroup$
    – cs0815
    Commented Oct 23, 2018 at 14:27

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.