Does Frequentist statistics still make sense when the experiment is not repeatable?

Question

Frequentist statistics is based on the idea that probability should be viewed objectively, that an event's probability is the limit of its relative frequency in many trials, and that probabilities can be found by a repeatable objective process.

Given this, does frequentist statistics still make sense if an experiment cannot be repeated in practice? To give a very simple example. Suppose I have a vase and I want to measure its mass $M$ using a somewhat unreliable scale that gives me readings of $X=M+\epsilon$ where $\epsilon \sim N(0,1)$. I obtain measurements $X_1,X_2,\dots,X_n$ on which I do frequentist inference (calculate the maximum likelihood, confidence intervals of $M$ etc.), all well and good. However, suppose then I accidentally break my vase and the pieces are not retrievable, then is frequentist inference still valid? The whole idea of confidence intervals for example relies on the idea that upon repeated experiments, a 95% confidence interval will capture the true value of the parameter 95% of the time, but if I can only ever do the experiment once, how is the confidence interval meaningful?

The above example is obviously very artificial and stupid but there are countless situations in the real world where repeated experiments or collections of the same datasets is not possible. For example, data that was specific to a particular time in the past, like if you're trying to infer the net CO2 emissions on earth in the year 2005 based on data collected in 2005. In those situations, how does frequentist statistics justify itself? Or should one concede to Bayesian inference when repeat experiments/recollection of data is not possible?

Answer 1

You have hit on the major challenge to the frequentist interpretation

As a preliminary observation, the usual examples of non-repeatable events used for this question relate to predictions of one-off things like the outcome of an election. Unlike your vase example, these are situations where the event of interest is not repeatable, even in principle. For example, if a polling firm was previously attempting to determine the probability that Obama would beat Romney in the 2012 US election, what is the frequentist interpretation of the probablity of this one-off event (in the non-repeatable context in which it occurs)? I will make reference to these kinds of events rather than your vase event, since they better capture the philosophical issue you are raising.

---

What you have hit on here is really the main philosophical challenge made against the frequentist interpretation of probability. We often wish to make probabilistic predictions in relation to one-off events that occur in a very specific context and which cannot be repeated in the same context. Much of the discipline of predictive analysis applies to one-off events and probability and statistics seems like a natural discipline to use to inform such prediction. For example, a polling company might want to determine the probability that Obama will beat Romney in the 2012 election. (Imagine asking this question in, say, late October of 2012, when we don't yet know the answer.) But what, if anything, would the "probability" of this one-off event mean in the frequentist paradigm? Even if people were to run the presidential election between Obama and Romney over and over again, after the first time it would never again occur under the same political context as the one that was of actual interest (and so it would not really be a repitition of the same election). So the question to the frequentist is: Can we use probability in such cases, and if so, how can it be accorded the frequentist interpretation?

Frequentists have generated two main answers to this challenge. One answer to this challenge is that we shouldn't use probability in such contexts and that the attempt to do so is illusory because the event under consideration is non-repeatable even in principle. This view delimits the application of probability and statistics and says that it should only be applied in a relatively narrow class of problems where we have events that are (at least in principle) repeatable. In that case it is possible to deploy the frequentist interpretation with respect to the infinite sequence of repititions that is (in principle) possible. Another answer to this challenge is to invoke the metaphysical hypothesis of a "multiverse" and say that any event that is a one-off event in our own universe is just one manifestation of an infinite set of outcomes under the same context that occur (or in principle could occur) in parallel universes. This view claims that any event is (in principle) repeatable by virtue of the fact that other outcomes were possible and that all possible outcomes occurs in "some universe". As you can see, these issues stray into philosophical territory and raise questions about the admissibility and sensibleness of speculating on unobserved (and unobservable) repititions of an experiment or event. There may be other answers I'm unfamiliar with, but they would probably be variations of these two general strategies (i.e., either delimit the scope of application of probability, or somehow assert that all events are repeatable in principle).

While I would encourage you to read more broadly on philosophy and probability to learn more about this topic, my own view is that neither of the above responses of the frequentist viewpoint are satisfactory. My view is that the frequentist approach to probability cannot explain probability in all contexts and so should not be seen as a valid basis for probability theory. The more sensible approach is to view probability as an epistemological concept --- a decision-making tool developed to assess uncertainty, subject to some important measurement and consistency desiderata. As I've noted in many other posts (see e.g., here and here), the frequentist interpretation is valid to the extent that it essentially just asserts the LLN --- all practitioners of all philosophical schools that use probability accept the LLN and thereby accept the frequentist interpretation in contexts where it applies.

Answer 2

Simple answer, yes the frequentist approach is still viable. We start with a (admittedly idealized) model. Take your vase example. Pre-sample, under the model we know that the confidence procedure with random sampling produces CIs covering the parameter 95% of the time. No actual repetitions are required. Also, if you look at a large number of independent 95% CIs from different problems then if their underlying models were valid (a big ask) approximately 95% would contain their respective parameter.

Regarding your real life example using past one-off data, the question is whether it is reasonable to treat the data as if it was a realization from an underlying model with random variables. For this to give valid frequentist inferences, you would also need to make sure that the choice of model by you was made before you saw the actual data.