How to answer critiques about the inapplicability of the framework of frequentist statistics to the real world?

Question

I often hear the argument that frequentist stats is useless or contorted because no event is precisely repeatable, let alone repeatable infinitely many times, and because there are no iid sequences in the real world, since "causal factors cause distributions shifts (?!)". Here is an example of the egregious use of this sort of arguments:

http://mitchgordon.me/math/2021/04/02/probability.html

Having applied frequentist stat with success to real world problems for years, I couldn't disagree more with the argument(s). Can you help me finding the weak points of the following arguments:

• frequentist statistics is not applicable to real-world problems, because no event is perfectly repeatable, let alone repeatable infinitely many times.
• events which appear repeatable, iid, random, such as the coin flip, are actually deterministic. BTW, I think here the author is mistaking the repeatability of a single coin flip with the repeatability of the whole sequence of coin flips, which is what frequentist statistics is actually concerned with, but I don't think this makes a big difference for what it regards this critique.

Answer 1

I think the issue with the arguments raised in the question is the naive realist philosophy of models apparently behind it.

If we model an experiment in a frequentist manner, what we do is that, when using the model, we treat the experiment as if it would be infinitely repeatable, with random outcomes the relative frequency of which stabilises for a growing number of observations.

The stated arguments seem to imply that this is only appropriate if the experiment really and objectively is of this kind. But a model is an idealisation. It seems quite clear that involving the exact physics of coin tossing would be a pointless effort when making predictions regarding, for example, how many heads you will observe in 1000 tosses. This is very easily possible assuming an i.i.d. frequentist model. Now obviously there is no guarantee that reality behaves like what is stated in the model. This can however (at least to some extent) be checked empirically, for example using the runs test to see whether sequences of heads and tails deviate from what is expected under independence. What can be validated in this way is not the truth of the model, but its fitness for the task for which it is used.

The model can in this way be used without requiring that what is formalised by the model is true in a naive realist sense. This may work well or not; we should not forget that we're dealing with an idealisation and we're making assumptions that may affect our conclusions from the model. Therefore the model and its assumptions need to be critically discussed, using knowledge of the situation as well as empirical checks, and rejected or updated if required. Sometimes the best use of the model is to enable the researcher to learn in which way it is violated.

Note that concepts such as "independence" and "identical repetition" are ultimately human constructs. Assuming "independence" means something like "any conceivable source of dependence is deemed unimportant by the observer", "identical repetition" means that "the observer perceives the repetitions as not different in any relevant respect". This involves judgements of the observer that can be challenged, discussed, and sometimes empirically falsified. The observer themselves may only make such judgements in a tentative fashion, being open to learn and adjust in case of falsification or strong doubts raised.

Another remark on model assumptions: Let's say we are interested in estimating a certain real quantity, and we have observations related to it. Assuming a certain frequentist model (such as "data i.i.d. exponentially distributed with unknown parameter") and identifying a parameter with what we are interested in in reality allows us to derive an estimator that is in a well defined sense optimal in the model framework. So we may use this estimator to estimate the quantity of interest in reality. Although the guaranteed optimality of the estimator requires the model to be true, putting up a model like this is a clever way to motivate a reasonable estimator in reality, and even giving an indication about the uncertainty using, say, a confidence interval, even without any guarantee that the model is true. What the model has done here is giving us a rationale, an idea, for what to do, and we can think of this as making sense as long as there are no specific objections against that model. As long as we don't have a better model, it is hard to argue that we can do better than that (although I'd find it desirable to interpret results acknowledging that the reason for using the model is not that we knew that it's true).

Answer 2

As to the first critique, it could be a critique of any and all branches of the sciences. There are no perfectly repeatable experiments. It isn't really possible to completely control any experiment. A meteor could strike the location of the experiment, for example.

Also, the ability to repeat an experiment is irrelevant. Most Frequentist inferences are of the form $\Pr(t(x)|\theta)$. Frequencies in that framework are a limiting form. If the model is true, then a p-value has meaning. The better critique would be "what happens when the model is not true?" That is a good critique because your null is usually the opposite of what you really believe to be true.

Frequentist frequencies are not probabilities in the colloquial sense. They are probabilities that provide guarantees. Except for exact tests, when you read that $p<.05$ it really does not mean that $p=.05$. It just guarantees that the false positive rate, if the null is true, will not exceed five percent over an infinite number of repetitions. However, as the number of repetitions becomes large enough, it will tend to converge.

It is true that one cannot do an infinite number of repetitions and it may be 100% wrong if you only do one sample. Nonetheless, it does provide a sensible way to make inferences and decisions. It allows you a method to control how often you will be made a fool of. It does not allow you to say this time is not the time I will be a fool.

The difficulty isn't in the math or the use but in the human need for there to be no false positives or negatives. The problem is in the human need for statistical significance to map perfectly to something being true and a lack of significance to something being false.

The second critique is a valid critique of any probabilistic methodology. It is probably true that Bayesian methods handle this critique a bit better because of the logic behind the construction of Bayesian methods.

If you need to be purist about that, then one could restrict the use of Frequentist methods to those cases where there truly is no prior knowledge or where the true null hypothesis of interest is a sharp null.

Let me illustrate the point.

You have a U.S. quarter that you are going to toss 50,000 times in a specially made vacuum chamber with a carefully constructed coin tossing machine. You want to determine if the coin is fair.

Even if you believe the coins to be "roughly fair", it is reasonable to discard that belief unless there really has been a controlled study of the fairness of U.S. coins. As a side note, a group of engineering students has done such a study.

The toss is totally deterministic and highly controlled. It is unclear how a Frequentist methodology would be disadvantaged here.

Now let us redo the experiment a bit.

Let us pretend that you and I are going to gamble money on the fairness of the coin. Indeed, I believe that the coin is so unfair that it will come up heads ten times in a row. You believe it is a fair coin. Prior to gambling any money, we will do a pilot study and have a third party toss the coin ten times. It comes up heads six out of ten times.

So I ask that you ante up 500:1 odds. I will toss the coin ten times.

Just before you do that a friend whispers in your ear that I apprenticed under my uncle who was a stage magician. Also, you are told that I was arrested, but not convicted, as working a number of street games like three-card Monte and coin games under the pseudonym Slick Eddy. Charges were dropped because, although I may have borne a striking resemblance to the alleged perpetrator, nobody was willing to come forward to identify me in a police lineup.

Wouldn't you prefer to incorporate that information with a Bayesian prior?

It is true that there is no such thing as a random coin toss. Any physicist, magician, or conman will tell you the same thing.

The Frequentist method would tell you that the entire procedure was not fair, after the fact, but it wouldn't allow you to incorporate all outside information. The Frequentist method is perfectly accurate by construction, but intrinsically less precise in the resulting estimators in this case.

The second argument is fittedness to purpose. Frequentist statistics are not a universal cure for all that ails mankind. They are a tool in a toolkit.

Let us flip the above example upside down.

Imagine that you truly do not possess any outside knowledge on something that you really must make a decision on. You do have the ability to collect a sample and you can use Bayesian or Frequentist statistics.

Frequentist statistics minimize the maximum amount of risk that you will be required to take. Bayesian methods do not. Frequentist methods, despite having no background information, provide guaranteed performance levels. In a state of true ignorance, that is a valuable thing to have. The Bayesian method cannot do that.

Answer 3

Having read the blog post, I think the author is saying that we shouldn't use randomness in models of the real world because the real world is not random, since everything (such as a coin flip) actually has a cause.

This makes probability theory the science of last resort. Only after truly exhausting your ability to investigate causal factors and processes should you indulge in probabilistic thinking. Doing otherwise is a cop-out, one that dangerously feels “scientific.”

However, I do not agree with this. I would say that probability is simply a way of quantifying what you don't know, or even just stating that you don't know something.

Treating something as random is the same as saying that you don't know it.

So if you insist (as some people do) on never using probability theory, then you are assuming that you know everything relevant to the problem under investigation, which seems to me to be even more of a cop-out than straight-up admitting that you don't know some things.

Whether your lack of knowledge is quantified as "what if things had gone some other way?" (frequentist) or "I'm not sure about the underlying state of the world in the first place" (Bayesian) isn't so important. What is important is that, whatever answer you come up with, you shouldn't be certain that it's correct. That's just crazy!

As Lewian says in the answer above, "All models are wrong, but some are useful" also applies to probability itself. Expressing uncertainty in terms of probabilities is a model of the real world, and it's often useful.