# When Bayesian and frequentist statistics give different answers, is there a way to empirically test which one corresponds more closely to reality?

For example for this problem:

You have a coin that when flipped ends up head with probability p and ends up tail with probability 1−p. (The value of p is unknown.)

Trying to estimate p, you flip the coin 14 times. It ends up head 10 times.

Then you have to decide on the following event: "In the next two tosses we will get two heads in a row."

Would you bet that the event will happen or that it will not happen?

Could one flip a bunch of different coins a million times and see what happens? (Or write a program to do so?)

EDIT: Some people were asking, I think, how probability theory could possibly be relevant when empirical data was available. For the above problem, I wrote a script which would flip a coin with a 1% chance of landing heads, a 2% coin, a 3% coin, etc. fourteen times; and if it came up heads ten out of those fourteen times, then it would flip the coin twice more and see whether it came up heads both times. This "experiment" is repeated a bunch of times and the percent of the time you get two heads, given that you've gotten ten out of fourteen heads, is around 48%, which is exactly the Bayesian answer given on the website above.

I tried not to make any assumptions about the nature of probability and just run the experiment the way it would happen "in the wild", if it were possible to run a million trials of this kind in real life, but I'm slightly worried that I made some inherently Bayesian assumption in the writing of the code.

Thoughts?

• if we knew what reality was there would be no need for a model, frequentist or bayesian. May 18, 2015 at 17:25
• Why would an empirical test be needed? All statistical procedures can be evaluated with perfect accuracy using mathematical reasoning. Any empirical test would only be a test of whether various underlying assumptions may be true, not of the statistical procedures.
– whuber
May 18, 2015 at 19:22
• Here is an author who wrote Python scripts to do simulations, in order to show the difference between frequentist and bayesian methods. Feb 3, 2017 at 13:24

#### You can test model performance empirically or against a simulation (but you should not confuse these two things with each other)

If you have a model that makes predictions of observable things (which both Bayesian and frequentist models do), you can make predictions about outcomes and check those predications against what actually happened. You can also construct measures of prediction accuracy and compare these between models to see which models do a better or worse job of predicting some set of outcomes in a particular situation. This can be done either with empirical data (i.e., real world data coming from some process or mechanism) or with simulations from a model (i.e., generating outcomes on a computer using a PRNG in a model). While both of these approaches may potentially be useful for certain purposes, you should not confuse empirical data with simulation --- simulated binary values on a computer are simulated binary values on a computer; they are not empirical coin flips.

When you determine the results of a predictive/inferential model on an underlying simulation, you are effectively looking at how well the inferential model performs on data from an underlying stipulated model. This is a purely theoretical exercise and there are relevant theoretical results that impact on this. In some cases it is possible to derive the results without any simulation just from the stipulated models. As whuber points out in the comments to your question, statistical models and inference/prediction methods are predicated on assumptions and the procedures are typically sensible (or indeed sometimes optimised) against the assumed model form. Consequently, looking at the accuracy of predictions/inferences from a model against simulated data is effectively a test of the "distance" between the assumptions of the inferential model and the assumptions of the model used for simulation.

Unlike in the case of simulations, when you determine the results of a predictive/inferential model on empirical data from an observed real-world mechanism or process, there is no known "model" of the comparison process. The accuracy of the predictions will typically reflect how close the assumptions of the inferential model are to the reality of the mechanism or process generating the data in real life. Empirical analysis of this kind can be useful as a study in whether or not particular statistical models adequately capture real-world mechanisms or processes that produce data.

In Bayesian approach as the sample size grows the influence of the prior diminishes. With an infinite sample you'll get the same results in frequentist and Bayesian approaches. With a finite sample, or small sample, there is no way to tell which approach is more accurate.

• Counterexample: the parameter of interest $y$ is a mean. The prior is uniform on the positive reals but the true value of $y$ is negative. The Bayesian solution will never coincide with the frequentist estimate.
– Sycorax
May 18, 2015 at 17:37
• @user777, should we can start a thread on dangers of improper priors? :) May 18, 2015 at 18:16
• @user777 - not a fair comparison. The parameter space for the frequentist estimator should be $(0, \infty)$ if the investigator is absolutely certain that the true parameter is $> 0$, and of course both frequentist and Bayesian estimates would go to 0 from above as the sample size increases. After all, we don't treat the mean of the Poisson as if negative values were possible in frequentist analysis (see any reporting of confidence intervals on same.) May 18, 2015 at 19:57
• @Aksakal This isn't an issue with improper priors. Some gamma prior has the same "problem," but is itself proper and strictly positive. I point this out to illustrate that there are some assumptions underlying your post which are not explicit.
– Sycorax
May 18, 2015 at 20:22
• Isn't the Frequentist "approach" just the Bayesian approach with an unstated prior? Why present them as two different approaches? Why not just say the Bayesianism makes the prior explicit? Jun 6, 2015 at 8:30

In comparing Bayesian and frequentist methods note that the frequentist approach is indirect and doesn't really give an "answer" in the same sense as a direct probability model such as a Bayesian model. Notes above have concentrated on a point estimate of the population mean, which is fine; also look at inferential quantities.

When the person viewing the results is using the prior that was used in the calculations, the Bayesian results have to be right.

See https://www.fharrell.com/post/pvalprobs/ for more about problems caused by the indirectness of the frequentist approach.

sometimes it is just imposible to do that much experiments, unlike the example of flipping a coin , it could be some medical issues, which is unaffordable to be repeated for many times, in this case frequencist based methods not work, the only choice is Bayesian