The key difference between Bayesian statistical inference and frequentist statistical methods concerns the nature of the unknown parameters that you are trying to estimate. In the frequentist framework, a parameter of interest is assumed to be unknown, but fixed. That is, it is assumed that in the population there is only one true population parameter, for example, one true mean or one true regression coefficient. In the Bayesian view of subjective probability, all unknown parameters are treated as uncertain and therefore are be described by a probability distribution. Every parameter is unknown, and everything unknown receives a distribution.
Consequently, in frequentist inference, you are primarily provided with a point estimate of the unknown but fixed population parameter. This is the parameter value that, given the data, is most likely in the population. An accompanying confidence interval tries to give you further insight into the uncertainty that is attached to this estimate. It is important to realize that a confidence interval simply constitutes a simulation quantity. Over an infinite number of samples taken from the population, the procedure to construct a (95%) confidence interval will let it contain the true population value 95% of the time. This does not provide you with any information on how probable it is that the population parameter lies within the confidence interval boundaries that you observe in your very specific and sole sample that you are analyzing.
In Bayesian analyses, the key to your inference is the parameter of interest’s posterior distribution. It fulfils every property of a probability distribution and quantifies how probable it is for the population parameter to lie in certain regions. On the one hand, you can characterize the posterior by its mode. This is the parameter value that, given the data and its prior probability, is most probable in the population. Alternatively, you can use the posterior’s mean or median. Using the same distribution, you can construct a 95% credibility interval, the counterpart to the confidence interval in frequentist statistics. Other than the confidence interval, the Bayesian counterpart directly quantifies the probability that the population value lies within certain limits. There is a 95% probability that the parameter value of interest lies within the boundaries of the 95% credibility interval. Unlike the confidence interval, this is not merely a simulation quantity, but a concise and intuitive probability statement. For more on how to interpret Bayesian analysis, check Van de Schoot et al. 2014.
This is a very well written https://www.rensvandeschoot.com/tutorials/brms-started/ blog on Bayesian modeling. I have read this paragraph maybe like 20 times and it feels like I haven't understood the difference between the two schools of statistics enough to appreciate Bayesian.I don't get the intuition or picture behind the difference between the credible and confidence interval as explained in this para.
Here, are a few things that are messy for me to process:
In the frequentist framework, a parameter of interest is assumed to be unknown, but fixed. Why is it fixed?I don't get the idea, why say mean or coefficient is considered fixed in the frequentist school? Is it because we consider the data to be fixed? If it was assumed under frequentists that the population has the same parameter, then why take many samples from the population in the first place? But, why not just measure a single person/point?
Every parameter is unknown, and everything unknown receives a distribution. Why? How is this different from the frequentist where the parameter is still unknown?Why does frequentist gives a point estimate of its unknown parameters(which can also be interpreted as a probability distribution)
Unlike the confidence interval, this is not merely a simulation quantity, but a concise and intuitive probability statement.
What does it mean to say that the confidence interval is merely a simulation? Aren't we also sampling the posterior via simulating random sample draws?
Other than the confidence interval, the Bayesian counterpart directly quantifies the probability that the population value lies within certain limits. There is a 95% probability that the parameter value of interest lies within the boundaries of the 95% credibility interval.
Why is one better than the other?
Could someone explain these points? With a mental/visual map to understand these points better?I'm sure these are very basic questions, and before someone points me to a book /blog, I want to clarify that I have gone/going through such resources, these are doubts the arises while going through them so..now referring more book /videos for these doubts doesn't make sense to me.