“I was just wondering whether anyone could give me a quick summary of their interpretation of Bayesian vs. Frequentist approach including Bayesian statistical equivalents of the Frequentist p-value and confidence interval. In addition, specific examples of where one method would be preferable to the other are appreciated.”
At a certain level, you basically have it. However, I spent some time thinking about your question and thought I would add an answer to it.
The first thing I will do is change some language. When I speak about the Frequentist perspective, I will use the words data and parameters. When I speak about the Bayesian perspective, I will use the words observables and unobservables. Do notice that they are not the same. For example, if you are missing a data point, it is an unobservable but not a parameter in the Frequentist sense. Likewise, if it is known that the variance of a process is equal to some value $k$, then it is an observable. One of the difficulties of talking across paradigms is that people try and use the same words for almost identical concepts but are not quite the same.
In the general sense, you are correct. In the Frequentist view of the world, there are fixed but unknown parameters. Those fixed parameters determine what data could be seen. The data that could be seen is the sample space. I would note that when I say “fixed,” I do not necessarily mean that $\sigma^2=k$, it could be that $\sigma^2=k*t$ or any other function or possibly relation. I do not require homoskedasticity or that a distribution is stationary. I am saying that at any instantaneous moment, there is a parameter of fixed value. Those parameters are the determiners of that which is instantaneously possible.
Note that the long-run frequencies do not follow from the observations or their nature but the fixed nature of the natural system that the frequencies represent. An essential and exceedingly helpful side effect is that the sampling distributions of many estimators can be known. In the real world, we are often dealing with estimators rather than raw frequencies. Indeed, the raw frequencies are often of little use.
It is a bit more challenging to discuss the Bayesian view of the world because it has multiple axiomatizations. In contrast, there is only one axiomatization of the Frequentist side, Kolmogorov’s. Usually, the differences do not matter, especially in applied work. Nonetheless, they can matter in theoretical work. Savage’s and de Finetti’s solutions to probability differ in some theoretical constructions, as do others. That can result in differing applied Bayesian models, particularly in the social sciences.
In addition to axiomatic differences, there can be differences in interpretation created by the silences in the math. Bayesian theory often does not require you to adopt a particular point of view, but we are humans, and we like to feel comfortable. I suspect people that work in quantum mechanics have the same difficulty.
As an example, there are two equally valid ways that you could view a prior distribution for an unobservable. The first would be that the unobserved quantity is fixed, but its location is unknown. The prior represents your uncertainty about its location. It is a representation of your beliefs about it. The second would be that Nature draws the actual value of the unobservable when you do an experiment from a distribution called the prior. Your prior represents your best estimate of what nature is doing. They are equivalent; to reject one is to deny the other. You can emotionally reject one, but not the math. You can assert one of the two, but that is a statement of emotional comfort, not math. The math doesn’t make nature do anything.
Randomness on the Bayesian side of the coin is uncertainty. There are unobservable things that you want to know more about, or to take actions about or because of, likely based on observable things. You are uncertain about the unobservables. You are certain of the observables. Please note that just because you are certain, treating the observables as fixed in the same sense as Frequentist parameters, does not imply that your observations are valid. If you observe a magician providing you with data, the observations are fixed. It does not mean that they are accurately informative of an underlying phenomenon.
Regardless, Bayesian probability measures and statistics are subjective. They depend entirely upon the prior knowledge of the system. A seasoned engineer with twenty years of experience and a graduate degree will have a different prior regarding soil samples for constructing a bridge than a fresh-out engineer who graduated engineering school last month. That difference in skill and knowledge can have very real-world consequences. A new engineer happily accepting the results of a t-test may find a grumpy senior engineer requiring more sampling and a rejection of the results by the inclusion of his or her prior.
Bayesian methods are about updating beliefs. The probability distributions are the distributions of belief. That may imply that there would be no scientific use, but that is not true. If one were to adopt the prior of an ardent opponent and show that even a highly bigoted opponent should assume the opposite belief, then that is very convincing. A passionate proponent of using Ivermectin to treat COVID, as long as they do not have a degenerate prior conviction that there is only one answer, will give up on Ivermectin as data comes through. It may take much longer than for a person with no personal opinion one way or the other, but it will happen. To be honest, because doctors prescribed so much Ivermectin in the last few months to keep their patients from going to other doctors, there is now an extensive data set. We have data from controlled experiments and natural experiments. The upshot of this is that people should get vaccinated and seek other treatments such as the monoclonal or polyclonal antibody infusions early in the infection.
As long as your prior beliefs are not degenerate, they can change upon seeing data, then the data will drive you to reality. Subject to you not living in the Truman show or gathering your entire worldview from magicians and con artists, the data wins eventually.
As to Bayesian equivalents to the p-value or the confidence interval, there are none. A p-value provides the probability of observing a result as extreme or more extreme if the null hypothesis is true. There can be no real Bayesian equivalent because there is no equivalent to the null hypothesis on the Bayesian side. No hypothesis is special, and there is no restriction to a null and an alternative. You can have any finite number of hypotheses that you find meaningful.
The closes thing is the Bayesian posterior probability. It is a statement of how much weight you give to the truth of a hypothesis. It has nothing to do with chance. The hypothesis is not assumed to be true. The question the posterior resolves is what probability do you give or how much credence or credibility do you provide a hypothesis.
There is no Bayesian equivalent to a confidence interval. A confidence interval is any function that guarantees that the interval will cover a parameter to some specified percentage of the time. There is an infinite number of such functions. Confidence intervals are not unique. Suppose you repeat an experiment an infinite number of times. In that case, the percentage of time that your interval will cover the parameter will never be less than your desired guaranteed percentage.
Of course, since infinite repetition isn’t feasible, it is just a model, as are Bayesian models.
Of great importance, if you perform an experiment and a parameter is estimated with 95% confidence to be in the interval $[a,b]$, that does not imply that there is a 95% chance that the parameter is in the interval. It is a statement of confidence in the interval building process. You believe that at least 95% of your intervals will cover the parameter as the number of experiments goes to infinity.
The closest Bayesian equivalent is the credible set. It is not an interval, and it does not have to be a connected set. It can certainly be true that the Frequentist confidence interval is $[5,15]$, when the Bayesian set of equivalent probability is $[6,7]\cup[8,9.5]$. The set can be disjoint if some area is improbable. As there is an infinite number of ways to subset a probability distribution, there is an endless number of possible credible sets that all add to at least some chosen percentage.
The credible interval is created by applying some rule to the posterior probability distribution. So a 95% credible set is the region where you would give at least a 95% probability of finding the parameter given your observations and prior knowledge.
Which method you should use depends entirely upon what usage you are going to put the techniques to.
Fisher’s method of maximum likelihood should be used whenever you want to acquire new knowledge. If you wonder if something is true, then you collect data and research it. Take that data, plug it into the method of maximum likelihood, and use that to generate a p-value or a likelihood ratio. If the p-value is small enough, then provisionally accept that your null hypothesis is false and do more research into the topic. If it is not small enough for you, then realize that you have wasted your time and go on to other, hopefully, more fruitful, things.
Pearson and Neyman’s frequency method should be used when it matters whether you accept or reject something. It allows you to create an acceptance and rejection region and gives you a way to control for statistical power. An excellent example of that would be quality control inspection. The method says that if you choose some value, $\alpha$, and stick to it, then you will be made a fool of no more often than $\alpha$ percentage of the time.
Laplace’s method of inverse probability, Bayesian analysis, should be used when you need to find something, gamble, take personal action, or update your beliefs. You should never ever place money at risk other than with Bayesian methods unless you want someone to have the ability to force you to take sure losses. Risk-taking with money is my area, so it colors my perspective. Likewise, if you need to find a downed plane, use a Bayesian method. If you need to find an unobservable quantity with observable data, Bayes is your tool. If you need to be able to make factual statements about a parameter using conventions that we can all agree with, Frequentist or Likelihoodhist methods are your toolkit.
That boils it down. The frequency side answers, “what are the minimal statements we can all agree with because we can have statistical confidence in the procedures that were used to create it?” The Bayesian side answers, “what should I believe, or how should I act, based only on what I saw and prior observations and the prior observations of other people that I have chosen to endorse?”