Is there any *mathematical* basis for the Bayesian vs frequentist debate?

Question

It says on Wikipedia that:

the mathematics [of probability] is largely independent of any interpretation of probability.

Question: Then if we want to be mathematically correct, shouldn't we disallow any interpretation of probability? I.e., are both Bayesian and frequentism mathematically incorrect?

I don't like philosophy, but I do like math, and I want to work exclusively within the framework of Kolmogorov's axioms. If this is my goal, should it follow from what it says on Wikipedia that I should reject both Bayesianism and frequentism? If the concepts are purely philosophical and not at all mathematical, then why do they appear in statistics in the first place?

Background/Context:
This blog post doesn't quite say the same thing, but it does argue that attempting to classify techniques as "Bayesian" or "frequentist" is counter-productive from a pragmatic perspective.

If the quote from Wikipedia is true, then it seems like from a philosophical perspective attempting to classify statistical methods is also counter-productive -- if a method is mathematically correct, then it is valid to use the method when the assumptions of the underlying mathematics hold, otherwise, if it is not mathematically correct or if the assumptions do not hold, then it is invalid to use it.

On the other hand, a lot of people seem to identify "Bayesian inference" with probability theory (i.e. Kolmogorov's axioms), although I'm not quite sure why. Some examples are Jaynes's treatise on Bayesian inference called "Probability", as well as James Stone's book "Bayes' Rule". So if I took these claims at face value, that means I should prefer Bayesianism.

However, Casella and Berger's book seems like it is frequentist because it discusses maximum likelihood estimators but ignores maximum a posteriori estimators, but it also seems like everything therein is mathematically correct.

So then wouldn't it follow that the only mathematically correct version of statistics is that which refuses to be anything but entirely agnostic with respect to Bayesianism and frequentism? If methods with both classifications are mathematically correct, then isn't it improper practice to prefer some over the others, because that would be prioritizing vague, ill-defined philosophy over precise, well-defined mathematics?

Summary: In short, I don't understand what the mathematical basis is for the Bayesian versus frequentist debate, and if there is no mathematical basis for the debate (which is what Wikipedia claims), I don't understand why it is tolerated at all in academic discourse.

Answer 1

Probability spaces and Kolmogorov's axioms

A probability space $\mathcal{P}$ is by definition a tripple $(\Omega, \mathcal{F}, \mathbb{P} )$ where $\Omega$ is a set of outcomes, $\mathcal{F}$ is a $\sigma$-algebra on the subsets of $\Omega$ and $\mathbb{P}$ is a probability-measure that fulfills the axioms of Kolmogorov, i.e. $\mathbb{P}$ is a function from $\mathcal{F}$ to $[0,1]$ such that $\mathbb{P}(\Omega)=1$ and for disjoint $E_1, E_2, \dots$ in $\mathcal{F}$ it holds that $P \left( \cup_{j=1}^\infty E_j \right)=\sum_{j=1}^\infty \mathbb{P}(E_j)$.

Within such a probability space one can, for two events $E_1, E_2$ in $\mathcal{F}$ define the conditional probability as $\mathbb{P}(E_1|_{E_2})\stackrel{def}{=}\frac{\mathbb{P}(E_1 \cap E_2)}{\mathbb{P}(E_2)}$

Note that:

1. this ''conditional probability'' is only defined when $\mathbb{P}$ is defined on $\mathcal{F}$, so we need a probability space to be able to define conditional probabilities.
2. A probability space is defined in very general terms (a set $\Omega$, a $\sigma$-algebra $\mathcal{F}$ and a probability measure $\mathbb{P}$), the only requirement is that certain properties should be fulfilled but apart from that these three elements can be ''anything''.

More detail can be found in this link

Bayes' rule holds in any (valid) probability space

From the definition of conditional probability it also holds that $\mathbb{P}(E_2|_{E_1})=\frac{\mathbb{P}(E_2 \cap E_1)}{\mathbb{P}(E_1)}$. And from the two latter equations we find Bayes' rule. So Bayes' rule holds (by definition of conditional probabilty) in any probability space (to show it, derive $\mathbb{P}(E_1 \cap E_2)$ and $\mathbb{P}(E_2 \cap E_1)$ from each equation and equate them (they are equal because intersection is commutative)).

As Bayes rule is the basis for Bayesian inference, one can do Bayesian analysis in any valid (i.e. fulfilling all conditions, a.o. Kolmogorov's axioms) probability space.

Frequentist definition of probability is a ''special case''

The above holds ''in general'', i.e. we have no specific $\Omega$, $\mathcal{F}$, $\mathbb{P}$ in mind as long as $\mathcal{F}$ is a $\sigma$-algebra on subsets of $\Omega$ and $\mathbb{P}$ fulfills Kolmogorov's axioms.

We will now show that a ''frequentist'' definition of $\mathbb{P}$ fulfills Kolomogorov's axioms. If that is the case then ''frequentist'' probabilities are only a special case of Kolmogorov's general and abstract probability.

Let's take an example and roll the dice. Then the set of all possible outcomes $\Omega$ is $\Omega=\{1,2,3,4,5,6\}$. We also need a $\sigma$-algebra on this set $\Omega$ and we take $\mathcal{F}$ the set of all subsets of $\Omega$, i.e. $\mathcal{F}=2^\Omega$.

We still have to define the probability measure $\mathbb{P}$ in a frequentist way. Therefore we define $\mathbb{P}(\{1\})$ as $\mathbb{P}(\{1\}) \stackrel{def}{=} \lim_{n \to +\infty} \frac{n_1}{n}$ where $n_1$ is the number of $1$'s obtained in $n$ rolls of the dice. Similar for $\mathbb{P}(\{2\})$, ... $\mathbb{P}(\{6\})$.

In this way $\mathbb{P}$ is defined for all singletons in $\mathcal{F}$. For any other set in $\mathcal{F}$, e.g. $\{1,2\}$ we define $\mathbb{P}(\{1,2\})$ in a frequentist way i.e. $\mathbb{P}(\{1,2\}) \stackrel{def}{=} \lim_{n \to +\infty} \frac{n_1+n_2}{n}$, but by the linearity of the 'lim', this is equal to $\mathbb{P}(\{1\})+\mathbb{P}(\{2\})$, which implies that Kolmogorov's axioms hold.

So the frequentist definition of probability is only a special case of Kolomogorov's general and abstract definition of a probability measure.

Note that there are other ways to define a probability measure that fulfills Kolmogorov's axioms, so the frequentist definition is not the only possible one.

Conclusion

The probability in Kolmogorov's axiomatic system is ''abstract'', it has no real meaning, it only has to fulfill conditions called ''axioms''. Using only these axioms Kolmogorov was able to derive a very rich set of theorems.

The frequentist definition of probability fullfills the axioms and therefore replacing the abstract, ''meaningless'' $\mathbb{P}$ by a probability defined in a frequentist way, all these theorems are valid because the ''frequentist probability'' is only a special case of Kolmogorov's abstract probability (i.e. it fulfills the axioms).

One of the properties that can be derived in Kolmogorov's general framework is Bayes rule. As it holds in the general and abstract framework, it will also hold (cfr supra) in the specific case that the probabilities are defined in a frequentist way (because the frequentist definition fulfills the axioms and these axioms were the only thing that is needed to derive all theorems). So one can do Bayesian analysis with a frequentist definition of probability.

Defining $\mathbb{P}$ in a frequentist way is not the only possibility, there are other ways to define it such that it fulfills the abstract axioms of Kolmogorov. Bayes' rule will also hold in these ''specific cases''. So one can also do Bayesian analysis with a non-frequentist definition of probability.

EDIT 23/8/2016

@mpiktas reaction to your comment:

As I said, the sets $\Omega, \mathcal{F}$ and the probability measure $\mathbb{P}$ have no particular meaning in the axiomatic system, they are abstract.

In order to apply this theory you have to give further definitions (so what you say in your comment "no need to muddle it further with some bizarre definitions'' is wrong, you need additional definitions).

Let's apply it to the case of tossing a fair coin. The set $\Omega$ in Kolmogorov's theory has no particular meaning, it just has to be ''a set''. So we must specify what this set is in case of the fair coin, i.e. we must define the set $\Omega$. If we represent head as H and tail as T, then the set $\Omega$ is by definition $\Omega\stackrel{def}{=}\{H,T\}$.

We also have to define the events, i.e. the $\sigma$-algebra $\mathcal{F}$. We define is as $\mathcal{F} \stackrel{def}{=} \{\emptyset, \{H\},\{T\},\{H,T\} \}$. It is easy to verify that $\mathcal{F}$ is a $\sigma$-algebra.

Next we must define for every event in $E \in \mathcal{F}$ its measure. So we need to define a map from $\mathcal{F}$ in $[0,1]$. I will define it in the frequentist way, for a fair coin, if I toss it a huge number of times, then the fraction of heads will be 0.5, so I define $\mathbb{P}(\{H\})\stackrel{def}{=}0.5$. Similarly I define $\mathbb{P}(\{T\})\stackrel{def}{=}0.5$, $\mathbb{P}(\{H,T\})\stackrel{def}{=}1$ and $\mathbb{P}(\emptyset)\stackrel{def}{=}0$. Note that $\mathbb{P}$ is a map from $\mathcal{F}$ in $[0,1]$ and that it fulfills Kolmogorov's axioms.

For a reference with the frequentist definition of probability see this link (at the end of the section 'definition') and this link.

Answer 2

Stats is not Math

First, I steal @whuber's words from a comment in Stats is not maths? (applied in a different context, so I'm stealing words, not citing):

If you were to replace "statistics" by "chemistry," "economics," "engineering," or any other field that employs mathematics (such as home economics), it appears none of your argument would change.

All these fields are allowed to exist and to have questions that are not solved only by checking which theorems are correct. Though some answers at Stats is not maths? disagree, I think it is clear that statistics is not (pure) mathematics. If you want to do probability theory, a branch of (pure) mathematics, you may indeed ignore all debates of the kind you ask about. If you want to apply probability theory into modeling some real-world questions, you need something more to guide you than just the axioms and theorems of the mathematical framework. The remainder of the answer is rambling about this point.

The claim "if we want to be mathematically correct, shouldn't we disallow any interpretation of probability" also seems unjustified. Putting an interpretation on top of a mathematical framework does not make the mathematics incorrect (as long as the interpretation is not claimed to be a theorem in the mathematical framework).

The debate is not (mainly) about axioms

Though there are some alternative axiomatizations*, the(?) debate is not about disputing Kolmogorov axioms. Ignoring some subtleties with zero-measure conditioning events, leading to regular conditional probability etc., about which I don't know enough, the Kolmogorov axioms and conditional probability imply the Bayes rule, which no-one disputes. However, if $X$ is not even a random variable in your model (model in the sense of the mathematical setup consisting of a probability space or a family of them, random variables, etc.), it is of course not possible to compute the conditional distribution $P(X\mid Y)$. No-one also disputes that the frequency properties, if correctly computed, are consequences of the model. For example, the conditional distributions $p(y\mid \theta)$ in a Bayesian model define an indexed family of probability distributions $p(y; \theta)$ by simply letting $p(y \mid \theta) = p(y; \theta)$ and if some results hold for all $\theta$ in the latter, they hold for all $\theta$ in the former, too.

The debate is about how to apply the mathematics

The debates (as much as any exist**), are instead about how to decide what kind of probability model to set up for a (real-life, non-mathematical) problem and which implications of the model are relevant for drawing (real-life) conclusions. But these questions would exist even if all statisticians agreed. To quote from the blog post you linked to [1], we want to answer questions like

How should I design a roulette so my casino makes $? Does this fertilizer increase crop yield? Does streptomycin cure pulmonary tuberculosis? Does smoking cause cancer? What movie would would this user enjoy? Which baseball player should the Red Sox give a contract to? Should this patient receive chemotherapy?

The axioms of probability theory do not even contain a definition of baseball, so it is obvious that "Red Sox should give a contract to baseball player X" is not a theorem in probability theory.

Note about mathematical justifications of the Bayesian approach

There are 'mathematical justifications' for considering all unknowns as probabilistic such as the Cox theorem that Jaynes refers to, (though I hear it has mathematical problems, that may or not have been fixed, I don't know, see [2] and references therein) or the (subjective Bayesian) Savage approach (I've heard this is in [3] but haven't ever read the book) that proves that under certain assumptions, a rational decision-maker will have a probability distribution over states of world and select his action based on maximizing the expected value of a utility function. However, whether or not the manager of Red Sox should accept the assumptions, or whether we should accept the theory that smoking causes cancer, cannot be deduced from any mathematical framework, so the debate cannot be (only) about the correctness of these justifications as theorems.

Footnotes

*I have not studied it, but I've heard de Finetti has an approach where conditional probabilities are primitives rather than obtained from the (unconditional) measure by conditioning. [4] mentions a debate between (Bayesians) José Bernardo, Dennis Lindley and Bruno de Finetti in a cosy French restaurant about whether $\sigma$-additivity is needed.

**as mentioned in the blog post you link to [1], there might be no clear cut debate with every statistician belonging to one team and despising the other team. I have heard it said that we are all pragmatics nowadays and the useless debate is over. However, in my experience these differences exist in, for example, whether someone's first approach is to model all unknowns as random variables or not and how interested someone is in frequency guarantees.

References

[1] Simply Statistics, a statistical blog by Rafa Irizarry, Roger Peng, and Jeff Leek, "I declare the Bayesian vs. Frequentist debate over for data scientists", 13 Oct 2014, http://simplystatistics.org/2014/10/13/as-an-applied-statistician-i-find-the-frequentists-versus-bayesians-debate-completely-inconsequential/

[2] Dupré, M. J., & Tipler, F. J. (2009). New axioms for rigorous Bayesian probability. Bayesian Analysis, 4(3), 599-606. http://projecteuclid.org/download/pdf_1/euclid.ba/1340369856

[3] Savage, L. J. (1972). The foundations of statistics. Courier Corporation.

[4] Bernardo, J.M. The Valencia Story - Some details of the origin and development of the Valencia International Meetings on Bayesian Statistics. http://www.uv.es/bernardo/ValenciaStory.pdf

Answer 3

The mathematical basis for the Bayesian vs frequentist debate is very simple. In Bayesian statistics the unknown parameter is treated as a random variable; in frequentist statistics it is treated as a fixed element. Since a random variable is a much more complicated mathematical object than a simple element of the set, the mathematical difference is quite evident.

However, it turns out that the actual results in terms of models can be surprisingly similar. Take linear regression for example. Bayesian linear regression with uninformative priors leads to a distribution of a regression parameter estimate, whose mean is equal to the estimate of the parameter of frequentist linear regression, which is a solution to a least squares problem, which is not even a problem from probability theory. Nevertheless the mathematics which was used to arrive at the similar solution is quite different, for the reason stated above.

Naturally because of the difference of treatment of the unknown parameter mathematical properties (random variable vs element of the set) both Bayesian and frequentist statistics hit on cases where it might seem that it is more advantageous to use a competing approach. Confidence intervals is a prime example. Not having to rely on MCMC to get a simple estimate is another. However, these are usually more matters of taste and not of mathematics.

Answer 4

I don't like philosophy, but I do like math, and I want to work exclusively within the framework of Kolmogorov's axioms.

How exactly would you apply Kolmogorov's axioms alone without any interpretation? How would you interpret probability? What would you say to someone who asked you "What does your estimate of probability $0.5$ mean?" Would you say that your result is a number $0.5$, which is correct since it follows the axioms? Without any interpretation you couldn't say that this suggests how often we would expect to see the outcome if we repeated our experiment. Nor could you say that this number tells you how certain are you about the chance of an event happening. Nor could you answer that this tells you how likely do you believe the event to be. How would you interpret expected value - as some numbers multiplied by some other numbers and summed together that are valid since they follow the axioms and a few other theorems?

If you want to apply the mathematics to the real world, then you need to interpret it. The numbers alone without interpretations are... numbers. People do not calculate expected values to estimate expected values, but to learn something about reality.

Moreover, probability is abstract, while we apply statistics (and probability per se) to real world happenings. Take the most basic example: a fair coin. In the frequentist interpretation, if you threw such a coin a large number of times, you would expect the same number of heads and tails. However, in a real-life experiment this would almost never happen. So $0.5$ probability has really nothing to to with any particular coin thrown any particular number of times.

Probability does not exist

-- Bruno de Finetti

Answer 5

My view of the contrast between Bayesian and frequentist inference is that the first issue is the choice of the event for which you want a probability. Frequentists assume what you are trying to prove (e.g., a null hypothesis) then compute the probability of observing something that you already observed, under that assumption. There is an exact analogy between such reverse-information flow-order probabilities and sensitivity and specificity in medical diagnosis, which have caused enormous misunderstandings and need to be bailed out by Bayes' rule to get forward probabilities ("post-test probabilities"). Bayesians compute the probability of an event, and absolute probabilities are impossible to compute without an anchor (the prior). The Bayesian probability of the veracity of a statement is much different from the frequentist probability of observing data under a certain unknowable assumption. The differences are more pronounced when the frequentist must adjust for other analyses that have been done or could have been done (multiplicity; sequential testing, etc.).

So the discussion of the mathematical basis is very interesting and is a very appropriate discussion to have. But one has to make a fundamental choice of forwards vs. backwards probabilities. Hence what is conditioned upon, which isn't exactly math, is incredibly important. Bayesians believe that full conditioning on what you already know is key. Frequentists more often condition on what makes the mathematics simple.

Answer 6

I will break this up into two separate questions and answer each.

1.) Given the different philosophical views of what probability means in a Frequentist and Bayesian perspective, are there mathematical rules of probability that apply to one interpretation and do not apply to another?

No. The rules of probability remain exactly the same between the two groups.

2.) Do Bayesians and Frequentists use the same mathematical models to analyze data?

Generally speaking, no. This is because the two different interpretations suggest that a researcher can gain insight from different sources. In particular, the Frequentist framework is often thought to suggest that one can make inference on the parameters of interest only from the data observed, while a Bayesian perspective suggests that one should also include independent expert knowledge about the subject. Different data sources means different mathematical models will be used for analysis.

It is also of note that there are plenty divides between the models used by the two camps that is more related to what has been done than what can be done (i.e. many models that are traditionally used by one camp can be justified by the other camp). For example, BUGs models (Bayesian inference Using Gibbs sampling, a name that no longer accurately describes the set of models for many reasons) are traditionally analyzed with Bayesian methods, mostly due to the availability of great software packages to do this with (JAGs, Stan for example). However, there is nothing that says these models must be strictly Bayesian. In fact, I worked on the project NIMBLE that builds these models in the BUGs framework, but allows the user much more freedom on how to make inference on them. While the vast majority of the tools we provided were customizable Bayesian MCMC methods, one could also use maximum likelihood estimation, a traditionally Frequentist method, for these models as well. Similarly, priors are often thought of as what you can do with Bayesian that you cannot do with Frequentist models. However, penalized estimation can provide for the same models using regularizing parameter estimates (although the Bayesian framework provides an easier way of justifying and choosing regularization parameters, while Frequentists are left with, in a best case scenario of a lots of data, "we chose these regularization parameters because over a large number of cross-validated samples, they lowered the estimated out of sample error"...for better or for worse).

Answer 7

Bayesians and Frequentists think probabilities represent different things. Frequentists think they're related to frequencies and only make sense in contexts where frequencies are possible. Bayesians view them as ways to represent uncertainty. Since any fact can be uncertain, you can talk about the probability of anything.

The mathematical consequence is that Frequentists think the basic equations of probability only sometimes apply, and Bayesians think they always apply. So they view the same equations as correct, but differ on how general they are.

This has the following practical consequences:

(1) Bayesians will derive their methods from the basic equations of probability theory (of which Bayes Theorem is just one example), while Frequentists invent one intuitive ad-hoc approach after another to solve each problem.

(2) There are theorems indicating that if you reason from incomplete information you had better use the basic equations of probability theory consistently, or you'll be in trouble. Lots of people have doubts about how meaningful such theorems are, yet this is what we see in practice.

For example, it's possible for real world innocent looking 95% Confidence Intervals to consist entirely of values which are provably impossible (from the same info used to derive the Confidence Interval). In other words, Frequentist methods can contradict simple deductive logic. Bayesian methods derived entirely from the basic equations of probability theory don't have this problem.

(3) Bayesian is strictly more general than Frequentist. Since there can be uncertainty about any fact, any fact can be assigned a probability. In particular, if the facts you're working on is related to real world frequencies (either as something you're predicting or part of the data) then Bayesian methods can consider and use them just as they would any other real world fact.

Consequently any problem Frequentist feel their methods apply to Bayesians can also work naturally. The reverse however is often not true unless Frequentists invent subterfuges to interpret their probability as a "frequency" such as, for example, imagining the multiple universes, or inventing hypothetical repetitions out to infinity which are never performed and often couldn't be in principle.

Answer 8

Question: Then if we want to be mathematically correct, shouldn't we disallow any interpretation of probability? I.e., are both Bayesian and frequentism mathematically incorrect?

Yes, and this is exactly what people do both in Philosophy of Science and in Mathematics.

1. Philosophical approach. Wikipedia provides a compendium of interpretations/definitions of probability.

2. Mathematicians are not safe. In the past, the Kolmogorovian school had the monopoly of probability: a probability is defined as a finite measure that assigns 1 to the whole space ... This hegemony is no longer valid since there are new trends on defininig probability such as Quantum probability and Free probability.

Answer 9

The bayes/frequentist debate is based on numerous grounds. If you are talking about mathematical basis, I don't think there is much.

They both need to apply various approximate methods for complex problems. Two examples are "bootstrap" for frequentist and "mcmc" for bayesian.

They both come with rituals/procedures for how to use them. A frequentist example is "propose an estimator of something and evaluate its properties under repeated sampling" while a bayesian example is "calculate probability distributions for what you don't know conditional on what you do know". There is no mathematical basis for using probabilities in this way.

The debate is more about application, interpretation, and ability to solve real world problems.

In fact, this is often used by people debating "their side" where they will use a specific "ritual/procedure" used by the "other side" to argue that the whole theory should be thrown away for theirs. Some examples include...

• using stupid priors (and not checking them)
• using stupid CIs (and not checking them)
• confusing a computational technique with the theory (bayes is not mcmc!! Same goes for equating cross validation with machine learning)
• talking about a problem with a specific application with one theory and not how the other theory would solve the specific problem "better"

Answer 10

So then wouldn't it follow that the only mathematically correct version of statistics is that which refuses to be anything but entirely agnostic with respect to Bayesianism and frequentism? If methods with both classifications are mathematically correct, then isn't it improper practice to prefer some over the others, because that would be prioritizing vague, ill-defined philosophy over precise, well-defined mathematics?

No. It does not follow. Individuals who are unable to feel their emotions are biologically incapable of making decisions, including decisions that appear to have only one objective solution. The reason is that rational decision making depends upon our emotional capacity and our preferences both cognitive and emotional. While that is scary, it is the empirical reality.

Gupta R, Koscik TR, Bechara A, Tranel D. The amygdala and decision making. Neuropsychologia. 2011;49(4):760-766. doi:10.1016/j.neuropsychologia.2010.09.029.

A person who prefers apples to oranges cannot defend this as it is a preference. Conversely, a person who prefers oranges to apples cannot defend this rationally as it is a preference. People who prefer apples will often eat oranges because the cost of apples is too great compared to the cost of oranges.

Much of the Bayesian and Frequentist debate, as well as the Likelihoodist and Frequentist debate, was centered around mistakes of understanding. Nonetheless, if we imagine that we have a person who is well trained in all methods, including minor or no longer used methods such as Carnapian probability or fiducial statistics, then it is only rational for them to prefer some tools over other tools.

Rationality only depends upon preferences; the behavior depends upon preferences and costs.

It may be the case that from a purely mathematical perspective that one tool is better than the other, where better is defined using some cost or utility function, but unless there is a unique answer where only one tool could work, then both the costs and the preferences are to be weighed.

Consider the problem of a bookie considering offering a complex bet. Clearly, the bookie should use Bayesian methods in this case as they are coherent and have other nice properties, but also imagine that the bookie has a calculator only and not even a pencil and paper. It may be the case that the bookie, with the use of his calculator and by keeping track of things in his head can calculate the Frequentist solution and has no chance on Earth to calculate the Bayesian. If he is willing to take the risk of being "Dutch Booked," and also finds the potential cost small enough, then it is rational for him to offer bets using Frequentist methods.

It is rational for you to be agnostic because your emotional preferences find that to be better for you. It is not rational for the field to be agnostic unless you believe that all people share your emotional and cognitive preferences, which we know is not the case.

In short, I don't understand what the mathematical basis is for the Bayesian versus frequentist debate, and if there is no mathematical basis for the debate (which is what Wikipedia claims), I don't understand why it is tolerated at all in academic discourse.

The purpose of academic debate is to bring light to both old and to new ideas. Much of the Bayesian versus Frequentist debate and the Likelihoodist versus Frequentist debate came from misunderstandings and sloppiness of thought. Some came from failing to call out preferences for what they are. A discussion of the virtues of an estimator being unbiased and noisy versus and estimator being biased and accurate is a discussion of emotional preferences, but until someone has it, it is quite likely that the thinking on it will remain muddy throughout the field.

I don't like philosophy, but I do like math, and I want to work exclusively within the framework of Kolmogorov's axioms.

Why? Because you prefer Kolmogorov's to Cox's, de Finetti's or Savage's? Is that preference sneaking in? Also, probability and statistics are not math, they use math. It is a branch of rhetoric. To understand why this may matter consider your statement:

if a method is mathematically correct, then it is valid to use the method when the assumptions of the underlying mathematics hold, otherwise, if it is not mathematically correct or if the assumptions do not hold, then it is invalid to use it.

This is not true. There is a nice article on confidence intervals and their abuse its citation is:

Morey, Richard ; Hoekstra, Rink ; Rouder, Jeffrey ; Lee, Michael ; Wagenmakers, Eric-Jan, The fallacy of placing confidence in confidence intervals, Psychonomic Bulletin & Review, 2016, Vol.23(1), pp.103-123

If you read the different potential confidence intervals in the article, each one is mathematically valid, but if you then evaluate their properties, they differ very substantially. Indeed, some of the confidence intervals provided could be thought of as having "bad" properties, though they meet all of the assumptions in the problem. If you drop the Bayesian interval from the list and focus only on the four Frequentist intervals, then if you do a deeper analysis as to when the intervals are wide or narrow, or constant, then you will find that the intervals may not be "equal" though each meets the assumptions and requirements.

It is not enough for it to be mathematically valid for it to be useful or, alternatively, as useful as possible. Likewise, it could be mathematically true, but harmful. In the article, there is an interval that is at its most narrow precisely when there is the least amount of information about the true location and widest when perfect knowledge or near perfect knowledge exists about the location of the parameter. Regardless, it meets the coverage requirements and satisfies the assumptions.

Math can never be enough.

Answer 11

The following is taken from my manuscript on confidence distributions - Johnson, Geoffrey S. "Decision Making in Drug Development via Confidence Distributions." arXiv preprint arXiv:2005.04721 (2020).

In the Bayesian framework the population-level parameter of interest is considered an unrealized or unobservable realization of a random variable that depends on the observed data. This premise has cause and effect reversed. In order to overcome this the Bayesian approach reinterprets probability as measuring the subjective belief of the experimenter. Another interpretation is that the unknown fixed parameter, say theta, was randomly selected from a known collection or prevalence of theta's (prior distribution) and the observed data is used to subset this collection, forming the posterior. The unknown fixed true theta is now imagined to have instead been randomly selected from the posterior. Every time the prior or posterior is updated the sampling frame from where we obtained our unknown fixed true theta under investigation must be changed. A third interpretation is that all values of theta are true simultaneously. The truth exists in a superposition depending on the evidence observed (think Schrodinger's cat). Ascribing any of these interpretations to the posterior allows one to make philosophical probability statements about hypotheses given the data. While the p-value is typically not interpreted in the same manner, it does show us the plausibility of a hypothesis given the data - the ex-post sampling probability of the observed result or something more extreme if the hypothesis is true. This does not reverse cause and effect.

To the Bayesian, probability is axiomatic and measures the experimenter. To the frequentist, probability measures the experiment and must be verifiable. The Bayesian interpretation of probability as a measure of belief is unfalsifiable. Only if there exists a real-life mechanism by which we can sample values of theta can a probability distribution for theta be verified. In such settings probability statements about theta would have a purely frequentist interpretation (see the second interpretation of the posterior above). This may be a reason why frequentist inference is ubiquitous in the scientific literature.

The interpretation of frequentist inference is straight forward for non-statisticians by citing confidence levels, e.g. 'We are 15.9% confident that theta is less than or equal to theta_0.' Of course to fully appreciate this statement of confidence one must more fully define the p-value as a frequency probability of the experiment if the hypothesis is true (think of the proof by contradiction structure a prosecutor uses in a court room setting, innocent until proven guilty). A Bayesian approach may make it easy for some to interpret a posterior probability, e.g. 'There is 17.4% Bayesian belief probability that theta is less than or equal to theta_0.' Of course to fully appreciate this statement one must fully define Bayesian belief and make it clear this is not a verifiable statement about the actual parameter, the hypothesis, nor the experiment. If the prior distribution is chosen in such a way that the posterior is dominated by the likelihood or is proportional to the likelihood, Bayesian belief is more objectively viewed as confidence based on frequency probability of the experiment. In short, for those who subscribe to the frequentist interpretation of probability, the confidence distribution summarizes all probability statements about the experiment one can make. It is a matter of correct interpretation given the definition of probability and what constitutes a random variable. The posterior remains an incredibly useful tool and can be interpreted as a valid asymptotic confidence distribution. The frequentist framework can easily incorporate historical data through a fixed-effect meta-analysis.