How seriously should I think about the different philosophies of statistics?

Question

I've just finished a module where we covered the different approaches to statistical problems – mainly Bayesian vs frequentist. The lecturer also announced that she is a frequentist. We covered some paradoxes and generally the quirks of each approach (long run frequencies, prior specification, etc). This has got me thinking – how seriously do I need to consider this? If I want to be a statistician, do I need to align myself with one philosophy? Before I approach a problem, do I need to specifically mention which school of thought I will be applying? And crucially, do I need to be careful that I don't mix frequentist and Bayesian approaches and cause contradictions/paradoxes?

Answer 1

I think that the main takeaway here is this: the mere fact that there are these different philosophies of statistics and disagreement over them implies that translating the "hard numbers" that one gets from applying statistical formulae into "real world" decisions is a non-trivial problem and is fraught with interpretive peril.

Frequently, people use statistics to influence their decision-making in the real world. For example, scientists aren't running randomized trials on COVID vaccines right now for funsies: it is because they want to make real world decisions about whether or not to administer a particular vaccine candidate to the populace. Although it may be a logistical challenge to gather up 1000 test subjects and observe them over the course of the vaccine, the math behind all of this is well-defined whether you are a Frequentist or a Bayesian: You take the data you gathered, cram it through the formulae and numbers pop out the other end.

However, those numbers can sometimes be difficult to interpret: Their relationship to the real world depends on many non-mathematical things – and this is where the philosophy bit comes in. The real world interpretation depends on how we went about gathering those test subjects. It depends on how likely we anticipated this vaccine to be effective a priori (did we pull a molecule out of a hat, or did we start with a known-effective vaccine-production method?). It depends on (perhaps unintuitively) how many other vaccine candidates we happen to be testing. It depends on etc., etc., etc.

Bayesians have attempted to introduce additional mathematical frameworks to help alleviate some of these interpretation problems. I think the fact that the Frequentist methods continue to proliferate shows that these additional frameworks have not been super successful in helping people translate their statistical computations into real world actions (although, to be sure, Bayesian techniques have led to many other advances in the field, not directly related to this specific problem).

To answer your specific questions: you don't need to align yourself with one philosophy. It may help to be specific about your approach, but it will generally be totally obvious that you are doing a Bayesian analysis the moment you start talking about priors. Lastly, though, you should consider all of this very seriously, because as a statistician it will be your ethical duty to ensure that the numbers that you provide people are used responsibly – because correctly interpreting those numbers is a hard problem. Whether you interpret your numbers through the lens of Frequentist or Bayesian philosophy isn't a huge deal, but interpretation of your numbers requires familiarity with the relevant philosophy.

Answer 2

A preliminary note on my nomenclature: As a preliminary matter, I note that I have never liked the terms "frequentist school" for the philosophy and set of methods it designates, and so I instead refer to this school of thought as "classical". Both Bayesians and classical statisticians agree entirely on the relevant theorems pertaining to the laws of large numbers, so both groups agree that the "frequentist" interpretation of probability holds under valid assumptions (i.e., an exchangeable sequence of values representing "repetition" of an experiment). All Bayesians are also "frequentists", in the sense that we accept the laws of large numbers and agree that probability corresponds to limiting frequency in appropriate circumstances. Since there is no real disagreement on the underlying laws of large numbers, I view it as silly to say that one group is a "frequentist" school and the other isn't.

---

This has got me thinking – how seriously do I need to consider this?

Others may disagree here, but my view is that if you want to be a good statistician, it is important to take foundational questions in the field seriously, and devote serious thinking to them during your training. Philosophical and methodological issues can seem far-removed from data analysis, but they are foundational issues that inform your choice of modelling methods and your interpretation and communication of results.

Learning something always invovles a trade-off (though not always against other learning!) so you will need to decide the appropriate trade-off between learning the philosophical and foundational issues in statistics, versus using your time for something else. This trade-off will depend on your specific aspirations, in terms of how detailed you want your knowledge of the subject to be. When training to be an academic in the field (i.e., when doing my PhD) I spent quite a lot of time reading philosophical papers on this subject, mulling over their implications, and having late-night drunken conversations on the topic with reluctant young ladies at university parties. My view now ---as a practicing academic--- is that this was time well spent.

If I want to be a statistician, do I need to align myself with one philosophy?

If you find one philosophy/methodology to be exclusively correct then you should align yourself entirely with that one philosophy/methodology. However, there are many statisticians who find some merit in each approach under different circumstances, or view one paradigm as philosophically correct, but difficult to apply in certain cases. In any case, it is not necessary to align yourself exclusively with one approach.

To be a good statistician, you should certainly understand the difference between the two paradigms and be capable of applying models in either paradigm. You should also have some sense of when a particular approach might be easier to apply to solving a particular problem. (For example, some "paradoxes" arise under classical methods that are easily resolved in Bayesian analysis. Contrarily, some modelling situations are difficult to deal with in Bayesian analysis, such as when we want to test a specific null hypothesis against a broad but vague alternative hypothesis.) In general, if you can enlarge your "toolkit" to be familiar with more methods and models, you will have a greater capacity to deploy effective methods in statistical problems.

Before I approach a problem, do I need to specifically mention which school of thought I will be applying?

This depends on context, but for general modelling purposes, no --- this will be obvious from the type of model and analysis you apply. If you apply a prior distribution to the unknown parameters and derive a posterior distribution, we will know you are doing a Bayesian analysis. If you treat the unknown parameters as "unknown constants" and use classical methods, we will know you you are using classical analysis. In good statistical writing you should explicitly state the model you are using (and maybe give references if you are writing an academic paper), and you might take this occasion to explicitly note if you are doing a Bayesian analysis, but even if you don't, it will be obvious.

Of course, if the problem you are approaching is a theoretical or philosophical problem (as opposed to a data analysis problem) then it may hinge upon the relevant interpretation of probability, and the consequent methodological paradigm. In such cases you should explicitly state your philosophical/methodological approach.

And crucially, do I need to be careful that I don't mix frequentist and Bayesian approaches and cause contradictions/paradoxes?

Unless you regard one of these methods to be totally invalid, such that it should never be used, it would stand to reason that it is okay to mix methods under appropriate circumstances. Again, understanding the strong and weak points of each paradigm will assist you in understanding when it is easier to apply one paradigm or the other.

In practical statistical work, it is quite common to see Bayesian analysis that has some classical methods applied for diagnostic purposes to test underlying assumptions. Usually this occurs when we want to test some assumption of a Bayesian model against a broad and vague alternative (i.e., where the alternative is not specified as a parametric model which is itself amenable to Bayesian analysis). For example, we might conduct a Bayesian analysis using a linear regression model, but then apply the Grubb's test (a classical hypothesis test) to test whether the assumption of normally distributed error terms is reasonable. Alternatively, we might conduct alternative Bayesian analyses using a set of different models, but then conduct cross-validation using classical methods. Perhaps there are some Bayesian "purists" who completely eschew classical methods, but they are rare. (This partly depends on the state of knowledge in the field of Bayesian analysis; as the field develops further and expands its boundaries, it has less and less need for supplementation by classical methods. Consequently, you should see this as contextual, based on the present state of development of Bayesian theory and related computational tools, etc.)

If you mix the two methods then you certainly need to be mindful of creating contradictions or "paradoxes" in your analysis, but obviously that is going to require you to have a good understanding of the two paradigms, which further behoves you to devote time to learning them.

Answer 3

I try to add something to the already existing answers that are worthwhile to read.

1. I do think that the foundations discussion touch basic questions that are important to think about as statisticians, particularly "what do we mean by probability?" Also understanding "inferential logic" when running, say, tests, confidence intervals, or computing posteriors, is crucial.

2. I also think it is important to know that issues go beyond the Bayesian/frequentist distinction. Particularly, there are different varieties of Bayesians, which have at least to some extent a different understanding of probabilities, mainly the radical subjectivists, so-called objective Bayesians, and people who prefer Bayesian reasoning about models and parameters, but however give models and parameters a frequentist meaning, called "falsificationist Bayes" in Sec. 5 of Gelman and Hennig (2017), where we try to give a reasonably "neutral" overview. Furthermore, there are concepts of "aleatory probabilities" (as opposed to epistemic, i.e., formalising subjective uncertainty) that are not directly connected to long run frequencies (often referred to as "propensities").

3. From my point of view, a key for understanding concepts in the foundations of statistics and probabilities is that we are generally dealing with mathematical models, and reality is different, i.e., there are no "true" frequentist probabilities to be found, and neither is there any "truly rational and correct reasoning" that is identical to the Bayesian model of it. Regardless of whether we work in a Bayesian or frequentist way (and which specific variety of these), we use models in order to make mathematical reasoning available for understanding phenomena in reality, which involves abstraction, simplification, and also, in one way or another, manipulation. We are using them as tools for thinking; they are adapted to our thinking, not in the first place to any reality outside our thoughts. For this reason, all kinds of practical issues (like issues with sampling schemes, measurements, missing values, unobserved confounders etc.) are important, to some extent for improving our models, and to some extent in order to understand the limits of what we can do with whatever approach.

4. This means that I personally believe that the different foundational approaches should not be seen as a "right" or "wrong" philosophy of statistics, and this also means that nobody needs to commit themselves to one of them only. Particularly, "epistemic" probabilities (as often but not always employed in subjectivist/"objective" Bayesian reasoning) model the uncertainty of either an individual or of science/humankind as a whole, whereas "aleatory" probabilities (as usually employed in frequentist reasoning) model the behaviour of data generating processes out there in the world. These are different, and one can well be interested in one thing regarding one research question and the other thing regarding another.

5. I do think though that "mixing them up" in the same study is problematic. When doing probability modelling, results come as probabilities (be it p-values, confidence levels, or posterior probabilities), so a consistent meaning should be used for all probabilities that occur in the same model. I think that the statisticians should be clear about what they mean when employing probabilities in given circumstances, and mixing often is done in such a way that this is unclear (particularly I see a lot of Bayesian work in which the likelihood is apparently interpreted in a frequentist manner, referring to really existing data generating processes, where no explanation is given what the prior probabilities are meant to express, even if sometimes related in a very rough fashion to some available knowledge).

6. However, I also think that there can be "legitimate mixing", for example in situations in which the prior distribution can be interpreted as itself being generated from a "real process" (e.g., of studies/problems of a similar kind), or when Bayesians, when applying consistently epistemic probabilities, are still interested in the frequentist properties of what they are doing, because they may find the logic of frequentist modelling ("what would happen if reality behaved according to frequentist model X") useful for learning about the implications of Bayesian methods. Sometimes priors can be introduced arguing that their introduction improves the frequentist characteristics of a method, rather than arguing that they appropriately express subjective or objective epistemic probabilities. So I think "mixing" requires a careful distinction between what the different probabilities mean in the different circumstances, clear understanding why one is used in one place and another in another place, and why they were brought together.

Answer 4

What you shouldn't do is align yourself with one in the sense of declaring one of them "right" and the other one "wrong". They are just two different viewpoints on the same thing, giving you alternative "tools of the trade". As an expert, you should be conversant in both. You may choose, for practical reasons, to specialize more in one than in the other. As an analogy, think of a chef who specializes in French cuisine, but can still whip up a nice Thai curry when the occasion calls for it.

Your question sounds like you might be afraid of the presence of a war of dogmas in statistics. These wars of dogmas happen in science every now and then, but I would argue that they are more of an artifact of the way humans do science, than based on real-world facts. The usual progress of dogma wars is: first there are two theories about a "hot" unsolved problem, then there is a popularity contest which fizzles out over a few decades, then the next generation of scientists discovers that both of the theories have merit and are not as mutually exclusive as presented at the height of the debate. You can find many good historical examples for this, e.g. keynesianist vs. neoclassicist economists from the early 20th century, or the question of whether environmental influences are heritable (often still oversimplified as darwinists vs. lamarckians).

Luckily, there is currently no such war of dogmas in the statistics community (and there wasn't one even when Bayes was alive), so you don't have to choose one camp and try to lead it to victory or go down fighting. A statistician declaring themselves as frequentist is, at best, trying to say "I prefer working with frequentist tools", and at worst, a snob who finds all other philosophies "not proper", imagine here again the French chef, maybe a pre-globalization one, who considers Thai food to be "inedible".

You should understand the "frequentist" (or "bayesian") declaration as an important signal about the flavor of information you will get from lessons or discussion with that person, and that's all about it.

Answer 5

Many people have given way better answers than I possibly could, but there are two things I wanted to add.

1. The field, hypothesis, and type of data you are working with can heavily influence which philosophy you use. The hypothesis "The mass of a neutron is 1.001 times the mass of a proton" definitely has a true or false answer. A frequentist approach would be very well suited to testing this hypothesis. Compare that to "Competition drives populations into different areas." This is not always true, but it is true many times. It is completely valid to interpret a Bayesian test of this hypothesis as how often it is true or how significant this effect is.

2. I believe that you should write out how you are going to analyze the data before ever looking at it. Whenever you decide to deviate from this plan, add an explanation for why before you do the new tests. This is a way to help you identify biases before they influence your work. Plus, if you store this document with an independent review board, you are almost immune to accusations of p-hacking.

Answer 6

The data itself is often the same for both approaches. In practice, the Bayesian or frequentist philosophies determine different estimators to analyze that data. Conversely, some estimators can be rationalized by either philosophy. Within each approach, modeling choices are needed to take the model to data, that can sometimes be tested for out-of-sample predictive accuracy. This is particularly true of "empirical Bayes" estimators that try to fit the hyperparameters using data.

For this reason, it is useful to think of the broad statistical properties of the estimators, regardless of their original rationale. I will mention two that are particularly salient:

(1) Admissibility: According to Wikipedia "an admissible decision rule is a rule for making a decision such that there is no other rule that is always "better" than it (or at least sometimes better and never worse)." Admissibility is a very basic criterion that rules out estimators that are clearly bad (e.g. calculating a mean with a single observation, discarding data for no reason, scaling the data in a weird way, etc.). It is well known that Bayes estimators are admissible, and hence have minimal guarantees.

What's interesting is that the set of admissible estimators can be very large. From a Bayesian point of view, different priors can induce different admissible estimators. This is analogous to the concept of "efficiency" in economics. Two allocations are efficient if they don't "waste" resources, but can use different inputs depending on the planner's preferences: there is an efficiency frontier. An agnostic frequentist might view the use of priors as a way to describe a class of admissible estimators, that impose different preferences over the weight given to new information.

(2) Regularization: A prior can also be viewed as a form of regularization (reducing the complexity) in predictive models. This facilitates the estimation of complicated models with small sample sizes relative to the number of parameters. For instance this article shows that Ridge (a form of penalized linear regression) can be motivated as a bayesian estimator with a normal prior, and the tuning parameter as a hyperparameter. Hence these can be viewed as different routes to regulate the bias/variance trade-offs. Similar analogies have been found for Lasso and other recently proposed high-dimensional methods.

There are other theoretical connections. For example, the Bernstein-von-Mises Theorem shows that the credible set of Bayesian parametric models can be close to frequentist confidence intervals in large samples.

As an agnostic practitioner you want to either design tests of validity (even as a thought experiment) that contain tangible, replicable metrics (e.g. out-of-sample MSE), that can help you decide between alternative estimators.