Comparison of frequentist methods (say, averaged over Monte Carlo simulations) and Bayesian method

Question

I have read a lot of questions with answers like this one, How do Bayesians verify their methods using Monte Carlo simulation methods?, which stated that Monte Carlo methods are not suitable for verifying Bayesian methods. I'm interested to look at intervals, and I understand that we do confidence intervals for frequentist methods and credible intervals for Bayesian methods.

Suppose that I have generated data 100 times for a Monte Carlo simulation study and obtain an averaged estimator with standard error and 95% confidence interval for frequentist methods. The issue is that I also have a Bayesian method to work with and as far as i know, Monte Carlo simulation may not be ideal in this case. What is the best way to do simulations for both Bayesian and frequentist methods?

Answer 1

In a sense Bayesian methods do not need to be validated other than assessing lack of fit (just as frequentist methods must consider lack of model fit). Model fit if often best assessed by using posterior predictive distributions to generate new dependent variable values and comparing the new distribution with what was observed for Y. But Monte-Carlo simulations are very useful for demonstrating how Bayes works and for learning about the differences between frequentist and Bayesian.

Bayesian simulations are different and are insightful. Consider a standard frequentist simulation: assume the unknown effect is zero and see how often you bring evidence (say p < 0.05) for an effect. Then simulated from an effect of say 20% and see how often the frequentist procedure detects an effect (power). Contrast with Bayes which is intended to uncover the true effect no matter what it is. Simulate from a smooth distribution of unknown effects. Run the Bayesian analysis oblivious to the true effect. Estimate a calibration curve just as we do in predictive modeling: relate the posterior probability of a positive effect to the probability the true effect being positive. Likewise you can make a calibration curve of a posterior mean vs. the true mean. You'll see that if the simulation distribution equals the analysis prior the calibration will be perfect even when you stop a study early upon observing a large mean.

All of this is demonstrated in detail, with R code, at https://www.fharrell.com/post/bayes-seq