# Does Frequentist statistics still make sense when the experiment is not repeatable?

Frequentist statistics is based on the idea that probability should be viewed objectively, that an event's probability is the limit of its relative frequency in many trials, and that probabilities can be found by a repeatable objective process.

Given this, does frequentist statistics still make sense if an experiment cannot be repeated in practice? To give a very simple example. Suppose I have a vase and I want to measure its mass $$M$$ using a somewhat unreliable scale that gives me readings of $$X=M+\epsilon$$ where $$\epsilon \sim N(0,1)$$. I obtain measurements $$X_1,X_2,\dots,X_n$$ on which I do frequentist inference (calculate the maximum likelihood, confidence intervals of $$M$$ etc.), all well and good. However, suppose then I accidentally break my vase and the pieces are not retrievable, then is frequentist inference still valid? The whole idea of confidence intervals for example relies on the idea that upon repeated experiments, a 95% confidence interval will capture the true value of the parameter 95% of the time, but if I can only ever do the experiment once, how is the confidence interval meaningful?

The above example is obviously very artificial and stupid but there are countless situations in the real world where repeated experiments or collections of the same datasets is not possible. For example, data that was specific to a particular time in the past, like if you're trying to infer the net CO2 emissions on earth in the year 2005 based on data collected in 2005. In those situations, how does frequentist statistics justify itself? Or should one concede to Bayesian inference when repeat experiments/recollection of data is not possible?

• Simple answer, Yes. We start with a (admittedly idealized) model. Take your vase example. Pre-sample, under the model we know that the confidence procedure with random sampling produces CIs covering the parameter 95% of the time. No actual repetitions are required. Also, if you look at a large number of independent 95% CIs from different problems then if their underlying models were valid (a big ask) approximately 95% would contain their respective parameters. Feb 1 at 21:50
• @SextusEmpiricus. The accepted answer in your linked question is an off-topic Bayesian commentary on p-values. Feb 1 at 23:35
• @GrahamBornholt When there are three close votes or a binding close vote (diamond moderator or someone with a gold badge for a question tag), that kind of comment automatically disappears, since the information is contained in the closure message.
– Dave
Feb 2 at 0:08
• @GrahamBornholt I'm not sure why you made this your business but it's me who first suggested the duplicate. If you read that thread carefully, you'll notice that it has two answers. And the second answer, although not the accepted one, explains nicely that's the theory is based on the repeatability of statistical procedure, not the repeatability of the data. Feb 2 at 0:52
• @dipetkov I’m not sure everyone has access to the closure messages. // I second your take that the second answer is a good one.
– Dave
Feb 2 at 0:55

#### You have hit on the major challenge to the frequentist interpretation

As a preliminary observation, the usual examples of non-repeatable events used for this question relate to predictions of one-off things like the outcome of an election. Unlike your vase example, these are situations where the event of interest is not repeatable, even in principle. For example, if a polling firm was previously attempting to determine the probability that Obama would beat Romney in the 2012 US election, what is the frequentist interpretation of the probablity of this one-off event (in the non-repeatable context in which it occurs)? I will make reference to these kinds of events rather than your vase event, since they better capture the philosophical issue you are raising.

What you have hit on here is really the main philosophical challenge made against the frequentist interpretation of probability. We often wish to make probabilistic predictions in relation to one-off events that occur in a very specific context and which cannot be repeated in the same context. Much of the discipline of predictive analysis applies to one-off events and probability and statistics seems like a natural discipline to use to inform such prediction. For example, a polling company might want to determine the probability that Obama will beat Romney in the 2012 election. (Imagine asking this question in, say, late October of 2012, when we don't yet know the answer.) But what, if anything, would the "probability" of this one-off event mean in the frequentist paradigm? Even if people were to run the presidential election between Obama and Romney over and over again, after the first time it would never again occur under the same political context as the one that was of actual interest (and so it would not really be a repitition of the same election). So the question to the frequentist is: Can we use probability in such cases, and if so, how can it be accorded the frequentist interpretation?

Frequentists have generated two main answers to this challenge. One answer to this challenge is that we shouldn't use probability in such contexts and that the attempt to do so is illusory because the event under consideration is non-repeatable even in principle. This view delimits the application of probability and statistics and says that it should only be applied in a relatively narrow class of problems where we have events that are (at least in principle) repeatable. In that case it is possible to deploy the frequentist interpretation with respect to the infinite sequence of repititions that is (in principle) possible. Another answer to this challenge is to invoke the metaphysical hypothesis of a "multiverse" and say that any event that is a one-off event in our own universe is just one manifestation of an infinite set of outcomes under the same context that occur (or in principle could occur) in parallel universes. This view claims that any event is (in principle) repeatable by virtue of the fact that other outcomes were possible and that all possible outcomes occurs in "some universe". As you can see, these issues stray into philosophical territory and raise questions about the admissibility and sensibleness of speculating on unobserved (and unobservable) repititions of an experiment or event. There may be other answers I'm unfamiliar with, but they would probably be variations of these two general strategies (i.e., either delimit the scope of application of probability, or somehow assert that all events are repeatable in principle).

While I would encourage you to read more broadly on philosophy and probability to learn more about this topic, my own view is that neither of the above responses of the frequentist viewpoint are satisfactory. My view is that the frequentist approach to probability cannot explain probability in all contexts and so should not be seen as a valid basis for probability theory. The more sensible approach is to view probability as an epistemological concept --- a decision-making tool developed to assess uncertainty, subject to some important measurement and consistency desiderata. As I've noted in many other posts (see e.g., here and here), the frequentist interpretation is valid to the extent that it essentially just asserts the LLN --- all practitioners of all philosophical schools that use probability accept the LLN and thereby accept the frequentist interpretation in contexts where it applies.

• Thank you for your very detailed answer! The reason I used the examples I did is that mass is an objective quantity that should have a true value and be inferable under the frequentist paradigm. The probability of winning a specific election on the other hand seems a lot more like a social construct and therefore seem to make more sense in a Bayesian framework. Would you say even in the case when the parameter is something like mass, the frequentist interpretation is still unsatisfactory if the experiment in principle can't be repeated (like mass of CO2 emissions on earth in 2005)? Feb 2 at 10:59

Simple answer, yes the frequentist approach is still viable. We start with a (admittedly idealized) model. Take your vase example. Pre-sample, under the model we know that the confidence procedure with random sampling produces CIs covering the parameter 95% of the time. No actual repetitions are required. Also, if you look at a large number of independent 95% CIs from different problems then if their underlying models were valid (a big ask) approximately 95% would contain their respective parameter.

Regarding your real life example using past one-off data, the question is whether it is reasonable to treat the data as if it was a realization from an underlying model with random variables. For this to give valid frequentist inferences, you would also need to make sure that the choice of model by you was made before you saw the actual data.

• Sorry can you elaborate a bit more on what you mean by 'you would also need to make sure that the choice of model by you was made before you saw the actual data'? Also a separate question if you don't mind, does the likelihood you've chosen have to be the 'correct' one in some sense for the inference to be valid? Feb 1 at 22:19
• Suppose you had a suite of possible models available to you, and you looked at the data first and noticed one fit very well. That could be more of a curve-fitting exercise, and can invalidate inferences based on the same data (I am sure some will object to this proviso). Your last question is unanswerable I think. How close is close enough, and remember the model is an idealized model anyway. In statistics, everything is provisional given our currently available knowledge. Feb 1 at 22:43