4
$\begingroup$

Within Statistics, I have seen the following applications of simulation:

  • Bayesian Sampling: When the Posterior Distribution can not be analytically integrated, we use MCMC algorithms (e.g. Metropolis-Hastings) to "simulate" points from the Posterior Distribution
  • Model Misspecification: Suppose I create a statistical model where I assume that the response variable has a certain distribution (e.g. Exponential Distribution) - I can simulate data from different distributions (e.g. Gamma Distribution) and see how "robust" my model is to conditions outside of what the model was created under

However, I keep hearing that it can be important to simulate data from the same model itself - for example, I fit a specific regression model on a specific dataset, I might need to simulate data from this regression model itself. I am trying to understand why this might be important and useful.

As an example, suppose there is a school with 100 students. I assume that the heights of these students follow a Normal Distribution. I then measure the height of all students, calculate the average height (e.g. "a") and the standard deviation (e.g. "b"). I now have a Normal Distribution(a,b).

If I simulate many samples from this Normal(a,b) - the average value of all these simulations should equal to "a". The way I see it, simulating numbers from this Normal(a,b) has not really provided me with any new information .

This is where my confusion is:

  • I can understand why its important to simulate data from a different model to test the robustness of your own model. As an example, if I design a bridge that I believe is well suited to design weights of "x KG" and windspeeds of "y KM/h" - I might be interested in seeing how this bridge behaves under different conditions, such as with weights of " x + z KG" and windspeeds of "y + w KM/h".
  • But why is it important to simulate data from the same regression model that you just fit - how can this result any new information that the you can't analytically infer from the model itself?

To reiterate my confusion one more time - I can simulate 1000 random samples from a Normal(a,b) and find out that the average value of these samples is close to "a".... but I could have just taken the Expected value of Normal(a,b) and determined that the average of many samples from this distribution should be close to "a". Thus, what is the importance of simulating from the same model itself?

Thanks!

Note:

  • Additional Reference: "When conducting simulation studies to evaluate the performance of new and existing statistical methods for analyzing survival data, one is required to simulate event times under a known data generating model. Similarly, one may need to simulate event times for the purpose of power calculations when designing new studies." (https://www.jstatsoft.org/article/view/v097i03)
  • However, when reading this quotation - it is not immediately clear to me why simulating data from the same model is required for goals such as "power calculations"
$\endgroup$
8
  • $\begingroup$ Additional Reference: jstatsoft.org/article/view/v097i03 $\endgroup$
    – stats_noob
    Feb 5 at 18:50
  • 1
    $\begingroup$ "When conducting simulation studies to evaluate the performance of new and existing statistical methods for analyzing survival data, one is required to simulate event times under a known data generating model. Similarly, one may need to simulate event times for the purpose of power calculations when designing new studies." $\endgroup$
    – stats_noob
    Feb 5 at 18:50
  • 1
    $\begingroup$ If I understand the reference correctly, it doesn't actually say that you should simulate from the assumed model when doing data analysis. Rather it looks like this is meant for methodological research; if you don't have (exact) theory, you need simulations to compare estimators and make recommendations. (Of course this doesn't make it invalid to do it also in practical situations, but I don't think this is what they have in mind.) It's correct of course that this is another use of simulated data, from the model of interest. $\endgroup$ Feb 5 at 22:26
  • 1
    $\begingroup$ "The way I see it, simulating numbers from this Normal(a,b) has not really provided me with any new information." It hasn't provided you with any new information, because you already knew that the distribution was normal. But if you're feeding the simulated data into a model that doesn't know that, then this model will view this as new information. $\endgroup$
    – Stef
    Feb 6 at 9:50
  • 2
    $\begingroup$ You start your story with explaining why sampling is useful, and then following that you add a twist where you say that it is unclear why sampling is useful. But why you think this is unclear does not become clear. $\endgroup$ Feb 6 at 13:35

6 Answers 6

7
$\begingroup$

I should start by saying that most people in most data analysis situations will not simulate data from the assumed model, so the majority view seems to be that you can well get away without doing it. "I keep hearing" isn't exactly an authoritative reference. ;-)

However, a major use is connected to the issue of potential model misspecification. Simulating data from the model and using visualisation techniques in particular, you can see whether and in what sense your real data set is similar or deviates systematically from data sets simulated from the model.

Otherwise, very often the theory behind the methods that you are using is not exact (standard mean estimation under Normal assumption being an exception), but rather asymptotic (sample size going to infinity) or in other ways approximate. Simulating from the model then allows you to explore the simulated variation/uncertainty and compare it with (approximative) theory. An example is variable selection in regression in a stepwise manner using p-values (I don't recommend to do that!) - there is no (exact) theory that nails down how it will behave in general, and simulating from the model is a very good way to see what the problem with it is, and how reliable (or not) it is.

$\endgroup$
2
  • $\begingroup$ +1 Your second paragraph is a more general form of my answer. $\endgroup$
    – Galen
    Feb 5 at 18:49
  • $\begingroup$ @ Christian Hennig: Thank you for your answer! I added a reference (please see update). Thank you! $\endgroup$
    – stats_noob
    Feb 5 at 18:52
10
$\begingroup$

As you've noted, simulating from the posterior is common in Bayesian modelling. It is also common to simulate from the prior, which is often called prior predictive sampling or prior predictive checking. The purpose of doing these simulations is to check that the model predicts scientifically justifiable outcomes even before it is updated with data. For example, I'm going to seriously doubt my prior is justifiable if it predicts that moose run at the speed of light with some non-negligible probability.

You can read more about prior predictive checking in Bayesian Workflow, or see some code in the pymc docs and stan docs.

$\endgroup$
3
  • 1
    $\begingroup$ @ Galen: Thank you for your answer! I have heard about "predictive prior checking" -but I did not mention it in my question as I was trying to keep it concise. $\endgroup$
    – stats_noob
    Feb 5 at 18:59
  • 1
    $\begingroup$ Spoken like someone who has never seen a moose run out in front of a car! $\endgroup$
    – Obie 2.0
    Feb 6 at 3:42
  • $\begingroup$ @Obie2.0 You should adjust your priors then b/c I actually have! :P $\endgroup$
    – Galen
    Aug 17 at 21:49
1
$\begingroup$

Resources

An answer that is quite a bit more practical is cost. If you are going to invest time and money into a research project that tax dollars are going to contribute to, it might be helpful to know that your modeling is actually accurate to a degree. If I get awarded 500k USD to conduct research on the survival rate of germ species in hot climates, I better have an idea of what I'm doing before I throw away every cent.

Assumption Checking

Part of it can also be for checking your own assumptions about statistical modeling. Perhaps I think that any sample size, so long as it has strong effects, is "good enough" to conduct a study. I can simulate truly random data to emulate what should be known about sample sizes.

Personal Interest

Simulating data is also just fun. I know quite little about simulation, but every small nugget I learn makes me fascinated by what info I derive from it.

$\endgroup$
1
$\begingroup$

Simulating from the same model that you use to analyze the data is useful for understanding the model and its limitations. To some extent it's a sanity check that the model does not have some serious issues, because a model that does not work very well when its assumptions are met is problematic. You may say:"But how could this ever be the case?" And the answer includes, but is not limited to:

  • You have messed up and something you assumed/derived/implemented is just not right.
  • If you simulate a huge amount of data, you should hope to get something close to the parameter values you simulated with back. If you don't there's a hint the first bullet might apply.
  • Parameter identifiability: I.e. the data you plan to collect/the experiment you do does not inform you at all - or at least not very well - about the parameter you are interested in. Example: randomized experiment with a placebo group and a single dose of a drug (let's say 50 mg), the outcome is wonderfully normally distributed and you fit a non-linear sigmoid Emax model assuming that $Y_i \sim \beta_0 + \beta_1 \text{dose}^h / (\text{dose}^h + \text{ED50}^h) + \epsilon_i$ with i.i.d. $\epsilon_i \sim N(0, \sigma^2)$. All assumption for your model are met, it is even the true data generating model, but with just one active dose the parameters are not identified. This may sound like a very specific situation (and it is), it's just pointing out that you may occasionally be in that kind of situation (and not realize it, e.g. you specify multiple levels of variability that cannot be distinguished).
  • Poor small sample performance (where "small sample" could occasionally be pretty large) e.g. due to using asymptotics. Example: Logistic regression fit using maximum likelihood with asymptotic standard errors (vs. exact logistic regression or similar) in case of 0 (or 1) events out of 100 (group 1) vs. 5 events out of 100 (group 2). You might not realize this is an issue, until you evaluate systematically what your model/its implementation does in these situations.
  • Some particular issues keep occurring that cause issues in defining what the maximum likelihood estimate would be (like complete separation in logistic regression).
  • Numeric optimization, convergence or sampling problems (that might be particularly common in some data situations).
  • Other numerical problems (e.g. covariates that are on very different scales cause issues, finite difference approximation makes a MCMC sampler not work when proper auto-differentiation would have worked etc.)
$\endgroup$
0
$\begingroup$

Parametric bootstrapping relies on simulating data from an estimated model. If it is difficult to work out confidence intervals analytically, then simulating datasets from your estimated model can give you an estimate of the sampling distribution of your estimates.

$\endgroup$
0
$\begingroup$

If I simulate many samples from this Normal(a,b) - the average value of all these simulations should equal to "a". The way I see it, simulating numbers from this Normal(a,b) has not really provided me with any new information.

In this example, simulation is not important. The distribution of Normal(a,b) is well known and can be estimated with several computational techniques.

Simulation is important when the distribution is not so easily computed.

However, I keep hearing that it can be important to simulate data from the same model itself

It is not clear where you got this from. It might relate to situations where the model is not so clearly computed as your normal distribution example.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.