Since we now have unlimited computation power (relative to historical norms), do we need to use statistical methods instead of simulations?

Question

Allen Downey wrote a blog post in 2016 titled "There is still only one test" which discusses the advantages of running simulations over traditional statistical tests. It argues that since computation is now tens of thousands of times faster compared to historical norms, traditional analytical methods in statistics no longer make sense. For example, when the t-test was discovered in 1908 and it was extremely useful as all computations were done by hand. But these days an iPhone can run statistical calculations ~10^11 times faster than a human could by hand in 1908.

To quote Downey:

These analytic methods were necessary when computation was slow and expensive, but as computation gets cheaper and faster, they are less appealing because:

1. They are inflexible: If you use a standard test you are committed to using a particular test statistic and a particular model of the null hypothesis. You might have to use a test statistic that is not appropriate for your problem domain, only because it lends itself to analysis. And if the problem you are trying to solve doesn't fit an off-the-shelf model, you are out of luck.

2. They are opaque: The null hypothesis is a model, which means it is a simplification of the world. For any real-world scenario, there are many possible models, based on different assumptions. In most standard tests, these assumptions are implicit, and it is not easy to know whether a model is appropriate for a particular scenario.

One of the most important advantages of simulation methods is that they make the model explicit. When you create a simulation, you are forced to think about your modeling decisions, and the simulations themselves document those decisions.

He also provides a hands-on example of such a simulation in this blog post. This makes me wonder:

1. With the computational power available today, why do we still run traditional statistical tests instead of creating explicit models and running millions of simulations to figure out the true p-value of an observation?
2. Are there still scenarios where traditional statistical methods are preferable to simulations?

Answer 1

I have coded permutation tests for linear mixed effects models that contained interactions, because some coauthors were concerned about the non-normality of residuals.

It was a horrendous amount of work. And very error-prone because of the complexity. I would absolutely not have wanted to debug my own code half a year later. I can only imagine the pain for someone else having to do that.

And the results - in terms of p values of F statistics - were extremely close to those from standard parametric tests.

I absolutely agree that it is an advantage if tests are made explicit through simulation and permutation. However, I would say that resampling based approaches can be at least as opaque as standard parametric methods. And while resampling is indeed more flexible than parametric methods (which are, actually, quite flexible), this flexibility is useful exactly in more complex situations where a resampling approach is very complicated to code. And harder yet to explain to non-statisticians.

Also, the vast majority of statistics is (likely) done by non-statisticians with more or less deep statistical learning: psychologists, STEM scientists, engineers. There are simply not enough statisticians and statistics-affine data scientists around. It's hard enough for non-statisticians to understand when to use what test. I believe it is unrealistic to expect them to understand statistics (and have the necessary programming and software development skills) deeply enough to routinely use resampling-based alternatives. Yes, this would absolutely be desirable. Just not realistic.

Answer 2

Stephan as usual has very good points (+1).

Apart from complexities with implementations, some other points include:

• Fairly standard modern methods like GLMMs are quite computationally intensive. Try running a Gamma GLMM on a moderately large data set... once. Even with dedicated functions, parallel processing and a modern CPU as of 2024, $1000$ repetitions of a model fit to $n > 10,000$ samples takes hours.
• I work in credit risk, where it is not unusual to have millions, or even tens of millions of rows. Even fairly simple methods can take a long time to bootstrap. This might still be worth it for a single, final run of a test, but trial and error is a pain when every step takes forever to run.
• Resampling is often already part of the workflow. Cross-validation, repeated runs of cross-validation, repeated runs of nested cross-validation... several layers deep, an extra $1000 \times$ computation time quickly adds up.

Answer 3

This is a reoccurring question in slightly different forms. See for example:

• Why are hypothesis tests still used when we have the bootstrap and central limit theorem?
• Three questions about the article "Ditch p-values. Use Bootstrap confidence intervals instead"
• Is Statistics less important in the era of big data than in old days?

The common aspect in these questions is people thinking that simulations can replace calculations.

It is misunderstanding the reason for using assumptions behind the calculations:

• Often, the problem is not a lack of computational power, and instead it is a lack of data. The assumptions make that we can do statistics with little data. Bootstrapping, using the emperical distribution function, will not help much if that emperical distribution function is not accurately describing the population distribution.
• Often, the computations are made such that we gain more understanding about the problem. A formula gives an insight into the underlying mechanism and the behaviour of the system when parameters change. With a simulation we may not know what happens if certain parameters change. For example, relationships between mean, variance, sample size, etc. can be revealed if we make a description of some outcome in terms of a formula instead of table with numbers from a simulation. By having the output in terms of a formula the relationships are clear.

---

I would say that the arguments and conclusions from the blog post are a bit misguided. It states that there is a distinction between implicit and explicit model assumptions and how this would be different with simulation methods, but it is not clear where this comes from. The simulation is not doing anything different except for applying the model in a different way (by using simulations instead of exact computations).

The nuts example in the link to the blog is a good example. The simulation of the distribution of a chi-squared statistic is just a way to avoid to do the calculations directly, but it is not making different assumptions.

I believe that such a problem has actually occurred here as a question before. It will involve correlated Poisson distributed variables with a conditional sum. This can be solved just like the derivation of the chi-squared test (Why does chi-square testing use the expected count as the variance?). While searching for a link to that question, here is a similar case: Probability of Winning a Prize from a raffle with Multiple Winners with Multiple Entries . There are some model assumptions and we can solve or approximate the equations or we can get an answer based on simulations. The image below shows the results from simulations and computations in that question/answer

The simulation and the computation generate the same picture. But, the computation is much more than just those numbers which the simulation gives as well; it also gives the answer as a formula $P(\text{no prize}) = \exp(-\alpha k_\text{entries})$ which makes us understand the problem much better.

Simulations can often help when equations are difficult to solve, but a simulation only gives you a number, it doesn't solve the equations with a formula as output.

Answer 4

Simulations are great! ...so long as the models driving the simulations are accurate. There are many systems in the world that we do not yet fully understand, or that our models are only accurate to some degree for. For example, although weather models continue to improve, we're a long ways away from accurately forecasting where and when next years' hurricanes are going to land. It would be impossible for us to accurately simulate that behaviour at current time, because the system is more complex and vast than we are capable of understanding well enough to accurately predict by simulation.

However, statistics remains useful in such cases. We may not know where and when hurricanes will precisely land, but we can take existing data and apply statistics to create a risk model for how likely a hurricane would be in a given time period for some location (or set of locations) based on a combination of that historical data and what limited models we do have.

Somebody far wiser than me (I couldn't tell you who, I think somebody on Reddit) once said something to the effect of "Statistics is the study of things we haven't figured out how to study yet". For everything we understand very well, simulations are going to tend to be much better than statistical models, but we must recognize that there is an awful lot that we still do not understand.

After all, "the system is complicated, our knowledge is limited" -Dr. Tong Yu, P.Eng

Answer 5

In addition to the other answers, sometimes your simulation has to be run an insane number of times if you want to have any hope of bringing the sample variance in your metrics down to workable levels.

As a simple example imagine you want to use simulation to assess how good difference grid search strategies are. Being a good experimentalist you decide you should probably fully randomize the grid—the starting point, the target, and clutter. Let’s keep it simple and say you have a 10x10 grid. Let’s focus on the clutter for now. Assume each grid cell can either contain or not contain clutter. Thus we can model the positions of all clutter on the board simultaneously with a single 100 digit binary number (one bit per grid cell). A binary number of this size has 1.2676506002×10³⁰ possible different values. If your simulation outcome is sensitive to the arrangement of clutter on the grid, this represents a potentially massive source of variability in your simulation, and you may need to run your simulation tens of thousands of times or more to get meaningful metrics. And this was just a toy problem.

In general if your simulation is on an n-D grid or in an n-D space for n greater than one, you may encounter this problem.

Answer 6

How many variables do you have?

If you have 3 interacting variables and you're injecting data without an understanding of the system, 100k iterations may only be 100 iterations of the "useful" region. If you have 6 interacting variables, that's 10 iterations, and you may not be able to infer a useful fit from that.