55
$\begingroup$

We already had a thread asking who are Bayesians and one asking if frequentists are Bayesians, but there was no thread asking directly who are frequentists? This is a question that was asked by @whuber as a comment to this thread and it begs to be answered. Do they exist (are there any self-identified frequentists)? Maybe they were just made-up by Bayesians who needed a scapegoat to blame when criticizing the mainstream statistics?

Meta-comment to the answers that were already given: As contrast, Bayesian statistics are not only defined in terms of using Bayes theorem (non-Bayesians also use it), nor about using subjectivist interpretation of probability (you wouldn't call any layperson saying things like "I bet the chance is less than 50:50!" a Bayesian) - so can we define frequentism only in terms of adopted interpretation of probability? Moreover, statistics $\ne$ applied probability, so should definition of frequentism be focused solely on the interpretation of probability?

$\endgroup$
  • 10
    $\begingroup$ I would still like to know what is a "Frequentist"? I have seen this term used as an apparent reference to "any non-Bayesian" as well as to someone who subscribes to a Frequentist philosophy of probability. I know there are many of the former but I'm not sure there are any living exponents of the latter! $\endgroup$ – whuber Aug 29 '16 at 19:47
  • 9
    $\begingroup$ Count me in as a frequentist, please. Let's unite! United we'll be strong to destroy Bayesian at the gate once a for all! $\endgroup$ – Aksakal Aug 29 '16 at 20:15
  • 10
    $\begingroup$ @Aksakal repeat it infinite number of times and I'll consider it significant ;) $\endgroup$ – Tim Aug 29 '16 at 20:20
  • 5
    $\begingroup$ Careful whom you call Bayesian. Per D.V. Lindley "There's no one less Bayesian than an empirical Bayesian". Many users of so-called Bayesian methods are not really Bayesian. Now with fast computers and MCMC everyone suddenly is a Bayesian? To many, it's just another calculation, like any other, not a deep-seated philosophy. Is the population disjointly divided, and only between Frequentists and Bayesian. How about someone without relevant education, what are they? Can there be a degree (or frequency) of Bayesianness? Maybe different on different days of week or types of analysis? $\endgroup$ – Mark L. Stone Aug 29 '16 at 21:23
  • 10
    $\begingroup$ Tim, I am afraid this thread might be affected by the same confusion as some related threads: namely, there are frequentist-vs-bayesian statistical methods and there are frequentist-vs-bayesian interpretations of probability. These are two different issues! Which one are you asking about? The confusion is obvious already now, in the comments to @fcop's answer; fcop talks about the definition/interpretation of probability, whereas you mentioned OLS and this is about statistical methods. $\endgroup$ – amoeba Aug 29 '16 at 23:33

10 Answers 10

35
$\begingroup$

Some existing answers talk about statistical inference and some about interpretation of probability, and none clearly makes the distinction. The main purpose of this answer is to make this distinction.


The word "frequentism" (and "frequentist") can refer to TWO DIFFERENT THINGS:

  1. One is the question about what is the definition or the interpretation of "probability". There are multiple interpretations, "frequentist interpretation" being one of them. Frequentists would be the people adhering to this interpretation.

  2. Another is statistical inference about model parameters based on observed data. There is a Bayesian and a frequentist approaches to statistical inference, and frequentists would be the people preferring to use the frequentist approach.

Now comes a speculation: I think there are almost no frequentists of the first kind (P-frequentists), but there are lots of frequentists of the second kind (S-frequentists).


Frequentist interpretation of probability

The question of what is probability is a subject of intense ongoing debate with 100+ years of history. It belongs to philosophy. I refer anybody not familiar with this debate to the Interpretations of Probability article in the Stanford Encyclopedia of Philosophy which contains a section on frequentist interpretation(s). Another very readable account that I happen to know of, is this paper: Appleby, 2004, Probability is single-case or nothing -- which is written in the context of foundations of quantum mechanics, but contains sections focusing on what probability is.

Appleby writes:

Frequentism is the position that a probability statement is equivalent to a frequency statement about some suitably chosen ensemble. For instance, according to von Mises [21, 22] the statement “the probability of this coin coming up heads is 0.5” is equivalent to the statement “in an infinite sequence of tosses this coin will come up heads with limiting relative frequency 0.5”.

This might seem reasonable, but there are so many philosophical problems with this definition that one hardly knows where to start. What is the probability that it will rain tomorrow? Meaningless question, because how would we have an infinite sequence of trials. What is the probability of the coin in my pocket coming up heads? A relative frequency of heads in an infinite sequence of tosses, you say? But the coin will wear off and the Sun will go supernova before the infinite sequence can be finished. So we should be talking about a hypothetical infinite sequence. This brings one to the discussion of reference classes etc. etc. In philosophy one does not get away so easily. And by the way, why should the limit exist at all?

Furthermore, what if my coin were to come up heads 50% of the time during the first billion of years but then would start coming up heads only 25% of the time (thought experiment from Appleby)? This means that $P(\mathrm{Heads})=1/4$ by definition. But we will always be observing $\mathrm{Frequency}(\mathrm{Heads})\approx 1/2$ during the next billion years. Do you think such a situation is not really possible? Sure, but why? Because the $P(\mathrm{Heads})$ cannot suddenly change? But this sentence is meaningless for a P-frequentist.

I want to keep this answer short so I stop here; see above for the references. I think it is really difficult to be a die-hard P-frequentist.

(Update: In the comments below, @mpiktas insists that it is because the frequentist definition is mathematically meaningless. My opinion expressed above is rather that the frequentist definition is philosophically problematic.)


Frequentist approach to statistics

Consider a probabilistic model $P(X\mid\theta)$ that has some parameters $\theta$ and allows to compute the probability of observing data $X$. You did an experiment and observed some data $X$. What can you say about $\theta$?

S-frequentism is the position that $\theta$ is not a random variable; its true values in Real World are what they are. We can try to estimate them as some $\hat \theta$, but we cannot meaningfully talk about probability of $\theta$ being in some interval (e.g. being positive). The only thing we can do, is to come up with a procedure of constructing some interval around our estimate such that this procedure succeeds in encompassing true $\theta$ with a particular long-run success frequency (particular probability).

Most of the statistics used in natural sciences today is based on this approach, so there certainly are lots of S-frequentists around today.

(Update: if you look for an example of a philosopher of statistics, as opposed to practitioners of statistics, defending S-frequentist point of view, then read Deborah Mayo's writings; +1 to @NRH's answer.)


UPDATE: On the relationship between P-frequentism and S-frequentism

@fcop and others ask about the relationship between P-frequentism and S-frequentism. Does one of these positions imply another one? There is no doubt that historically S-frequentism was developed based on P-frequentist stance; but do they logically imply one another?

Before approaching this question I should say the following. When I wrote above that there are almost no P-frequentists I did not mean that almost everybody is P-subjective-bayesian-a-la-de-finetti or P-propensitist-a-la-popper. In fact, I believe that most statisticans (or data-scientists, or machine-learners) are P-nothing-at-all, or P-shut-up-and-calculate (to borrow Mermin's famous phrase). Most people tend to ignore foundation problems. And it is fine. We do not have a good definition of free will, or intelligence, or time, or love. But this should not stop us from working on neuroscience, or on AI, or on physics, or from falling in love.

Personally, I am not a S-frequentist, but neither do I have any coherent view on foundations of probability.

In contrast, almost everybody who did some practical statistical analysis is either a S-frequentist or a S-Bayesian (or perhaps a mixture). Personally, I published papers containing $p$-values and I have never (so far) published papers containing priors and posteriors over model parameters so this makes me a S-frequentist, at least in practice.

It is therefore clearly possible to be a S-frequentist without being a P-frequentist, despite what @fcop says in his answer.

Okay. Fine. But still: Can a P-bayesian be a S-frequentist? And can a P-frequentist be a S-bayesian?

For a convinced P-bayesian it is probably atypical to be a S-frequentist, but in principle entirely possible. E.g. a P-bayesian can decide that they do not have any prior information over $\theta$ and hence adopt a S-frequentist analysis. Why not. Every S-frequentist claim can certainly be interpreted with P-bayesian interpretation of probability.

For a convinced P-frequentist to be S-bayesian is probably problematic. But then it is very problematic to be a convinced P-frequentist...

$\endgroup$
  • 2
    $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – Glen_b Sep 5 '16 at 23:17
13
+500
$\begingroup$

Kolmogorov's work on Foundations of Theory of Probability has the section called "Relation to Experimental Data" on p.3. This is what he wrote there:

enter image description here

enter image description here

He's showing how one could deduct his Axioms by observing experiments. This is quite a frequentist way of interpreting the probabilities.

He has another interesting quote for impossible events (empty sets):

enter image description here

So, I think that if you're comfortable with these arguments, then you must admit that you're a frequentist. This label is not exclusive. You can be bi-paradigmous (I made up the word), i.e. both a frequentist and Bayesian. For instance, I become Bayesian when applying stochastic methods to phenomena which are not inherently stochastic.

UPDATE As I wrote earlier on CV, Kolmogorov's theory itself is not frequentist per se. It's as compatible with Bayesian view as with frequentist view. He put this cute footnote to the section to make very clear that he's abstaining from philosophy:

enter image description here

$\endgroup$
  • 1
    $\begingroup$ well most axiomatic systems are derived by ''abstracting'' or ''generalizing'' real world concepts. The same holds for the concept (axioms) of a ''metric'', they looked at the real world concept (eucildean distance) and ''found'' three properties that they used as ''axioms'' in a metric space. But your answer is good (+1), it shows that Kolmogorov did not ''invent'' his axioms ''out of the air'', he developed an axiomatic system that was linked to the understanding of what propability was at that time. $\endgroup$ – user83346 Aug 29 '16 at 20:54
  • 1
    $\begingroup$ Could you look at @mpiktas 's comments to this question and say that Kolmogorov himself gave that defintion ? see comments on stats.stackexchange.com/questions/230415/… $\endgroup$ – user83346 Aug 29 '16 at 21:09
  • 2
    $\begingroup$ @fcop, I remember his answer. It seemed reasonable. You can definitely derive Bayes axioms, and Kolmogorov shows that in the book. $\endgroup$ – Aksakal Aug 29 '16 at 21:16
  • 1
    $\begingroup$ +1 Exactly what I had in mind (the footnote, in particular) when I was writing comments to related threads. Also last comment by @Aksakal. $\endgroup$ – Graeme Walsh Aug 29 '16 at 23:03
  • 1
    $\begingroup$ @Aksakal: referring to your comment above: it was not ''his'' answer but mine :-) $\endgroup$ – user83346 Aug 31 '16 at 12:34
8
$\begingroup$

I believe that it is relevant to mention Deborah Mayo, who writes the blog Error Statistics Philosophy.

I won't claim to have a deep understanding of her philosophical position, but the framework of error statistics, as described in a paper with Aris Spanos, does include what is regarded as classical frequentist statistical methods. To quote the paper:

Under the umbrella of error-statistical methods, one may include all standard methods using error probabilities based on the relative frequencies of errors in repeated sampling – often called sampling theory or frequentist statistics.

And further down in the same paper you can read that:

For the error statistician probability arises not to measure degrees of confirmation or belief (actual or rational) in hypotheses, but to quantify how frequently methods are capable of discriminating between alternative hypotheses and how reliably they facilitate the detection of error.

$\endgroup$
  • 2
    $\begingroup$ +1, Well done mentioning Mayo's blog. She is indeed the most vocal philosopher of statistics who is supporting frequentist statistics. Most of the other rants on the internet these days are in favour of Bayesian statistics. $\endgroup$ – amoeba Sep 5 '16 at 22:39
6
$\begingroup$

Referring to this thread and the comments on it I think that the frequentists are those that define ''probability'' of an event as the long run relative frequency of the occurence of that event. So if $n$ is the number of experiments and $n_A$ the number of occurences of event $A$ then the probability of the event $A$, denoted by $P(A)$, is defined as $$P(A):=\lim_{n\to +\infty} \frac{n_A}n$$.

It is not hard to see that this definition fulfills Kolmogorov's axioms (because taking limits is linear, see also Is there any *mathematical* basis for the Bayesian vs frequentist debate?).

In order to give such a definition they must ''believe'' that this limit exists. So the frequentists are those who believe in the existence of this limit.

EDIT on 31/8/2016: on the distintion between S- and P-frequentism

As @amoeba distinguishes in his answer between S-frequentists and P-frequentists, where P-frequentists are the type of frequentists that I define supra, and as he also argues that it is hard to be a P-frequentist I added an EDIT section to argue that the opposite is true;

I argue that all S-frequentists are P-frequentists.

In the S-frequentism section @amoeba says ''this procedure succeeds in encompassing true $\theta$ with a particular long-run success frequency (particular probability).''

In his answer he also states that P-frequentists are a rare species.

But this ''long-run success frequency'', used to define S-frequentism, is what he defines as P-frequentism as it is the interpretation of $P(\widehat{CI} \ni \theta)$.

Therefore, according to his defintions every S-frequentist is also a P-frequentist. Therefore I conclude that P-frequentists are not so rare as argued by amoeba.

There is even more; @amoeba also argues that the S-frequentists consider the unknown parameter $\theta$ as fixed or non-random, therefore one can not talk about ''probability of $\theta$ having a particluar value'', he says that

''The only thing we can do, is to come up with a procedure of constructing some interval around our estimate such that this procedure succeeds in encompassing true $\theta$ with a particular long-run success frequency (particular probability).''

May I ask what might be the origin of the name ''frequentist'' : (a) the ''non-random $\theta$''-idea or (b) the ''long-run frequency''-idea ?

May I also ask @mpiktas who writes in his comment to the answer of amoeba:

'' It is very hard to be a P-frequentist, because it is practically impossible to give mathematically sound definition of such probability ''

If you need a defintion of P-frequentism to define the S-frequentism, how can one then be more S-frequentist than P-frequentist ?

$\endgroup$
  • 3
    $\begingroup$ @Tim: why would OLS be ''frequentist'' ? $\endgroup$ – user83346 Aug 29 '16 at 20:14
  • 2
    $\begingroup$ If it is so, then the whole discussion of who are frequentists does not seem to have much sense since their scope of interest is very limited and huge areas of statistics are out of their scope... (Including many areas that are of interest of Bayesians.) What I am saying is that this does not seem to be the definition since it is too narrow. $\endgroup$ – Tim Aug 29 '16 at 20:30
  • 2
    $\begingroup$ What do Bayesians (self-proclaimed, or merely pragmatists who are wearing their "Bayesian hat" for the interlude) believe is $\lim_{n\to +\infty} \frac{n_A}{n}$? $\endgroup$ – Silverfish Aug 29 '16 at 20:38
  • 1
    $\begingroup$ @Silverfish $lim_{n→+∞}n_A/n$ Bayesians are free to use long run frequencies as probabilities if they wish to do so, it is only frequentists that have no option. The key difference is that a Bayesian might say that $p_H \equiv lim_{n→+∞}n_A/n$, but in inference we are interested in defining a distribution over the plausible values of $p_H$ that we obtain from the 1E12 observations. $\endgroup$ – Dikran Marsupial Aug 30 '16 at 11:29
  • 3
    $\begingroup$ foundation of probability theory precisely for that reason, i.e. to give certain interpretations of probability sound mathematical background. Now you want to go back and "reinvent" probability with your "frequentist" definition. I find this misleading, hence I downvote your answers. You are welcome to ignore this, this is the way site works, there is always possibility for disagreement. $\endgroup$ – mpiktas Aug 30 '16 at 16:51
4
$\begingroup$

Really interesting question!

I'd put myself in the frequentist camp when it comes to understanding and interpreting probability statements, although I am not quite so hard-line about the need for an actual sequence of iid experiments to ground this probability. I suspect most people who don't buy the thesis that "probability is a subjective measure of belief" would also think about probability this way.

Here's what I mean: take our usual "fair" coin, with assignment $P(H)=0.5$. When I hear this, I form an image of someone tossing this coin many times and the fraction of heads approaches $0.5$. Now, if pressed, I would also say that the fraction of heads in any random sample from a finite sequence of such coin tosses will also approach $0.5$ as the sample size grows (independence assumption).

As has been stated by others, the biggest assumption is that this limit exists and is correct (i.e., limit is $0.5$), but I think just as importantly is the assumption that the same limit exists for randomly chosen sub-samples as well. Otherwise, our interpretation only has meaning wrt the entire infinite sequence (e.g., we could have strong autocorrelation that gets averaged out).

I think the above is pretty uncontroversial for frequentists. A Bayesian would be more focused on the experiment at hand and less on the long run behavior: they would state that their degree of belief that the next toss will be heads is $P(H) = 0.5$...full stop.

For a simple case such as coin tossing, we can see that the frequentist and Bayesian approaches are functionally equivalent, albeit philosophically very different. As Dikran Marsupial has pointed out, the Bayesian may in fact be utilizing the fact that empirically we see coins come up heads about as often as we see them come up tails (long run/large sample frequency as a prior).

What about things that cannot possibly have long run frequencies? For example, what is the probability North Korea will start a war with Japan in the next 10 years? For frequentists, we are really left in the lurch, since we cannot really describe the sampling distributions required to test such a hypothesis. A Bayesian would be able to tackle this problem by placing probability distribution over the possibilities, most likely based on eliciting expert input.

However, a key question comes up: where do these degrees of belief (or assumed value for the long run frequency) come from? I'd argue from psychology and say that these beliefs (especially in areas far from experimental data) come from what is referred to as the availability heuristic and representativness heuristic. There are slew of others that likely come into play. I argue this because in the absence of data to calibrate our beliefs (towards the observed long run frequency!), we must rely on heuristics, however sophisticated we make them seem.

The above mental heuristic thinking applies equally to Frequentists and Bayesians. What is interesting to me is that regardless of our philosophy, at the root, we place more belief in something that we think is more likely to be true, and we believe it to be more likely to be true because we believe there are more ways for it to be true, or we imagine that the pathways leading to it being true would happen more often (frequently:-) than those that would make it not true.

Since it's an election year, let's take a political example: What belief would we place in the statement "Ted Cruz will propose a ban assault rifles in the next 4 years". Now, we do have some data on this from his own statements, and we'd likely place our prior belief in the truth of this statement very near zero. But why? Why does his prior statements make us think this way? Because we think that highly ideological people tend to "stick to their guns" more than their pragmatist counterparts. Where does this come from? Likely from the studies done by psychologists and our own experiences with highly principled people.

In other words we have some data and the belief that for most cases where someone like Cruz could change their mind, they will not (again, a long-run or large-sample assessment of sorts).

This is why I "caucus" with the frequentists. It's not my dislike of Bayesian philosophy (quite reasonable) or methods (they're great!), but that if I dig deep enough into why I hold beliefs that lack strong large-sample backing, I find that I am relying on some sort of mental model where outcomes can be tallied (if implicitly) or where I can invoke long-run probabilities in a particular sub-process (e.g., Republicans vote against gun control measures X% of the time) to weight my belief one way or another.

Of course, this not really true frequentism, and I doubt that there are many people who subscribe to the von Mieses-esque interpretation of probability to the letter. However, I think it shows the underlying compatibility between Bayesian and Frequentist probability: Both are appealing to our inner heuristics regarding availability or what I call the "Pachinko" principle about frequencies along a chain of causation.

So perhaps I should call myself an "availabilist", to indicate that I assign probabilities based on how often I can imagine an event occurring as the outcome of a chain of events (with some rigor/modelling of course). If I have a lot of data, great. If I don't, then I will try to decompose the hypothesis into a chain of events and use what data I have (anecdotal or "common sense", as need be) to assess how often I would imagine such an event to occur.

Sorry for the longish post, great question BTW!

$\endgroup$
  • 1
    $\begingroup$ You make huge oversimplifications about subjectivist approach to probability. I agree that psychology plays role in how you work with data, but you can make subjectively-biased analysis in frequentist approach as well. Some Bayesians argue that in Bayesian case at least you make your subjective beliefs explicit, rather then ignoring them. $\endgroup$ – Tim Aug 30 '16 at 14:00
  • 1
    $\begingroup$ @Tim I wasn't saying that Frequentists are more objective (see my first part about the major assumptions for frequentists.) Also, what I was pointing out is that both frequentists and Bayesians ground their probabilities in their beliefs about how often something would occur, or how many things need to happen (and their individual likelihood for doing so) to make something true. $\endgroup$ – user75138 Aug 30 '16 at 14:03
  • 1
    $\begingroup$ @Tim since this is not a Post about Bayesianism vs Frequentism per se, I doubt I gave either their due...that would take a lot of page space! Fundamentally through, we all rely on mental heuristics when thinking about probability, and these heuristics lead us think of some types of event being able to happen "in more ways" or being "more accessible in a causal netowork" than others. $\endgroup$ – user75138 Aug 30 '16 at 14:05
  • 1
    $\begingroup$ @Tim OK...I added in some langugae to clarify when I transition from my (brief) discussion of Bayesianism to my main spiel. $\endgroup$ – user75138 Aug 30 '16 at 14:08
4
$\begingroup$

As @amoeba noticed, we have frequentist definition of probability and frequentist statistics. All the sources that I have seen until now say that frequentist inference is based on the frequentist definition of probability, i.e. understanding it as limit in proportion given infinite number random draws (as already noticed by @fcop and @Aksakal quoting Kolmogorov)

$$ P(A) = \lim_{n\to\infty} \frac{n_A}{n} $$

So basically, there is a notion of some population that we can repeatably sample from. The same idea is used in frequentist inference. I went through some classic papers, e.g. by Jerzy Neyman, to track the theoretical foundations of frequentist statistics. In the 1937 Neyman wrote

(ia) The statistician is concerned with a population, $\pi$, which for some reason or other cannot be studied exhaustively. It is only possible to draw a sample from this population which may be studied in detail and used to form an opinion as to the values of certain constants describing the properties of the population $\pi$. For example, it may be desired to calculate approximately the mean of a certain character possessed by the individuals forming the population $\pi$, etc.
(ib) Alternatively, the statistician may be concerned with certain experiments which, if repeated under apparently identical conditions, yield varying results. Such experiments are called random experiments [...]
In both cases described, the problem with which the statistician is faced is the problem of estimation. This problem consists in determining what arithmetical operations should be performed on the observational data in order to obtain a result, to be called an estimate, which presumably does not differ very much from the true value of the numerical character, either of the population $\pi$, as in (ia), or of the random experiments, as in (ib). [...]
In (ia) we speak of a statistician drawing a sample from the population studied.

In another paper (Neyman, 1977), he notices that the evidence provided in the data need to be verified by observing the repeated nature of the studied phenomenon:

Ordinarily, the 'verification', or 'validation' of a guessed model consists in deducing some of its frequentist consequences in situations not previously studied empirically, and then in performing appropriate experiments to see whether their results are consistent with predictions. Very generally, the first attempt at verification is negative: the observed frequencies of the various outcomes of the experiment disagree with the model. However, on some lucky occasions there is a reasonable agreement and one feels the satisfaction of having 'understood' the phenomenon, at least in some general way. Later on, invariably, new empirical findings appear, indicating the inadequacy of the original model and demanding its abandonment or modification. And this is the history of science!

and in yet another paper Neyman and Pearson (1933) write about random samples drawn from fixed population

In common statistical practice, when the observed facts are described as "samples," and the hypotheses concern the "populations", for which the samples have been drawn, the characters of the samples, or as we shall term them criteria, which have been used for testing hypotheses, appear often to be fixed by happy intuition.

Frequentist statistics in this context formalize the scientific reasoning where evidence are gathered, then new samples are drawn to verify the initial findings and as we accumulate more evidence our state of knowledge crystallizes. Again, as described by Neyman (1977), the process takes the following steps

(i) Empirical establishment of apparently stable long-run relative frequencies (or 'frequencies' for short) of events judged interesting, as they develop in nature.
(ii) Guessing and then verifying the 'chance mechanism', the repeated operation of which produces the observed frequencies. This is a problem of 'frequentist probability theory'. Occasionally, this step is labeled 'model building'. Naturally, the guessed chance mechanism is hypothetical.
(iii) Using the hypothetical chance mechanism of the phenomenon studied to deduce rules of adjusting our actions (or 'decisions') to the observations so as to ensure the highest 'measure' of 'success'. [...] the deduction of the 'rules of adjusting our actions' is a problem of mathematics, specifically of mathematical statistics.

Frequentists plan their research having in mind the random nature of data and the idea of repeated draws from fixed population, they design their methods based on it, and use it to verify their results (Neyman and Pearson, 1933),

Without hoping to know whether each separate hypothesis is true or false, we may search for rules to govern our behavior with regard to them, in following which we insure that, in the long run of experience, we shall not be too often wrong.

This is connected to repeated sampling principle (Cox and Hinkley, 1974):

(ii) Strong repeated sampling principle
According to the strong repeated sampling principle, statistical procedures are to be assessed by their behaviour in hypothetical repetitions under the same conditions. This has two facets. Measures of uncertainty are to be interpreted as hypothetical frequencies in long run repetitions; criteria of optimality are to be formulated in terms of sensitive behaviour in hypothetical repetitions.
The argument for this is that it ensures a physical meaning for the quantities that we calculate and that it ensures a close relation between the analysis we make and the underlying model which is regarded as representing the "true" state of affairs.

(iii) Weak repeated sampling principle
The weak version of the repeated sampling principle requires that we should not follow procedures which for some possible parameter values would give, in hypothetical repetitions, misleading conclusions most of the time.

As contrast, when using maximum likelihood we are concerned with the sample that we have, and in Bayesian case we make inference based on the sample and our priors and as new data appears we can perform Bayesian updating. In both cases the idea of repeated sampling is not crucial. Frequentists rely only on the data they have (as noticed by @WBT), but keeping in mind that it is something random and it is to be thought as a part of process of repeated sampling from the population (recall, for example, how confidence intervals are defined).

In frequentist case the idea of repeated sampling enables us to quantify the uncertainty (in statistics) and enables us to interpret real-life events in terms of probability.


As a side note, notice that neither Neyman (Lehmann, 1988), nor Pearson (Mayo, 1992) were as pure frequentists as we could imagine they were. For example, Neyman (1977) proposes using Empirical Bayesian and Maximum Likelihood for point estimation. On another hand (Mayo, 1992),

in Pearson's (1955) response to Fisher (and elsewhere in his work) is that for scientific contexts Pearson rejects both the low long-run error probability rationale [...]

So it seems that it is hard to find pure frequentists even among the founding fathers.


Neyman, J, and Pearson, E.S. (1933). On the Problem of the Most Efficient Tests of Statistical Hypotheses. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 231 (694–706): 289–337.

Neyman, J. (1937). Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability. Phil. Trans. R. Soc. Lond. A. 236: 333–380.

Neyman, J. (1977). Frequentist probability and frequentist statistics. Synthese, 36(1), 97-131.

Mayo, D. G. (1992). Did Pearson reject the Neyman-Pearson philosophy of statistics? Synthese, 90(2), 233-262.

Cox, D. R. and Hinkley, D. V. (1974). Theoretical Statistics. Chapman and Hall.

Lehmann, E. (1988). Jerzy Neyman, 1894 - 1981. Technical Report No. 155. Department of Statistics, University of Califomia.

$\endgroup$
  • 1
    $\begingroup$ +1 (already some days ago). Thanks for writing this up, there are some good quotes here and the list of references is very useful. In turn, I have now updated my answer in several places but mostly in the end, to clarify the relationship between P- and S-frequentism as I see it. I certainly agree that historically S-freq was built on P-freq ideas, as you argue here, but I don't think that they necessarily entail one another. $\endgroup$ – amoeba Sep 5 '16 at 22:56
4
$\begingroup$

Let me offer an answer that connects this question with a matter of current and very practical importance -- Precision Medicine -- while at the same time answering it literally as it was asked: Who are frequentists?

Frequentists are people who say things such as [1] (emphasis mine):

What does a 10% risk of an event within the next decade mean to the individual for whom it was generated? Contrary to what is thought, this risk level is not that person’s personal risk because probability is not meaningful in an individual context.

Thus, frequentists interpret 'probability' in such a way that it has no meaning in a singular context like that of an individual patient. My PubMed Commons comment on [1] examines the contortions its frequentist authors must undergo to recover a semblance of a probability-like notion applicable to the care of an individual patient. Observing how and why they do this may prove very instructive as to who is a frequentist. Also, the largely unilluminating subsequent exchange in the JAMA Letters section [2,3] is instructive as to the importance of explicitly recognizing limitations in frequentist notions of probability and attacking them directly as such. (I regret many CV users may find that [1] lies behind a paywall.)

The excellent and highly readable book [4] by L. Jonathan Cohen would repay the efforts of anyone interested in the OP's question. Of note, Cohen's book oddly was cited by [1] in connection with the claim "probability is not meaningful in an individual context," although Cohen clearly rebukes this view as follows [4,p49]:

Nor is it open to a frequency theorist to claim that all important probabilities are indeed general, not singular. It often seems very important to be able to calculate the probability of success for your own child’s appendectomy...


1] Sniderman AD, D’Agostino Sr RB, and Pencina MJ. “The Role of Physicians in the Era of Predictive Analytics.” JAMA 314, no. 1 (July 7, 2015): 25–26. doi:10.1001/jama.2015.6177. PubMed

2] Van Calster B, Steyerberg EW, and Harrell FH. “RIsk Prediction for Individuals.” JAMA 314, no. 17 (November 3, 2015): 1875–1875. doi:10.1001/jama.2015.12215. Full Text

3] Sniderman AD, D’Agostino Sr RB, and Pencina MJ. “RIsk Prediction for Individuals—reply.” JAMA 314, no. 17 (November 3, 2015): 1875–76. doi:10.1001/jama.2015.12221. Full Text

4] Cohen, L. Jonathan. An Introduction to the Philosophy of Induction and Probability. Oxford : New York: Clarendon Press ; Oxford University Press, 1989. Link to scanned pages 46-53 & 81-83

$\endgroup$
  • 1
    $\begingroup$ 2] is available online: lirias.kuleuven.be/bitstream/123456789/513795/1/… $\endgroup$ – Tim Aug 30 '16 at 20:47
  • $\begingroup$ I like the quote "risk can be thought of as the subjective level to which one “believes in” or is “pre- pared to bet on” the occurrence of a disease or event, 2 just as one bets on future 1-time events in games of sport". Thanks for your input and references. $\endgroup$ – Tim Aug 30 '16 at 20:53
  • 2
    $\begingroup$ Looks like a good answer, but I'm a little confused. I believe frequentists are concerned with procedures (experiments) which could, in theory, be run an infinite number of times. Doesn't mean they can't make a prediction about the next coin toss, just that their concerns/assurances concentrate on the process. True, they would say that "the US will win the World Cup this year" isn't something they could say anything about, but couldn't they say something about your child's appendectomy -- it is unique in once sense, but appendectomies on children are not. $\endgroup$ – Wayne Aug 30 '16 at 21:20
  • $\begingroup$ @Tim, thanks for the link, which includes both [2] and [3]. I have updated the text of my answer accordingly. $\endgroup$ – David C. Norris Aug 30 '16 at 21:46
  • 2
    $\begingroup$ @Wayne, I would suggest that amoeba's answer above--with its distinction between statistical procedures customarily associated with the 'frequency view of probability' and that frequency view itself--answers your comment. People might use frequentist statistical procedures, and yet interpret and use the resulting probabilities in fiducial or subjectivist ways. One need only look at the perpetual error of interpreting P-values as 'the probability the null hypothesis is true' to see that this is so. $\endgroup$ – David C. Norris Aug 30 '16 at 21:52
3
$\begingroup$

"Frequentists vs. Bayesians" from XKCD (under CC-BY-NC 2.5), click to discuss:

'Detector! What would the Bayesian statistician say if I asked him whether the--' [roll] 'I AM A NEUTRINO DETECTOR, NOT A LABYRINTH GUARD. SERIOUSLY, DID YOUR BRAIN FALL OUT?' [roll] '... yes.'

The general point of the frequentist philosophy illustrated here is a belief in drawing conclusions about the relative likelihood of events based solely ("purely") on the observed data, without "polluting" that estimation process with pre-conceived notions about how things should or should not be. In presenting a probability estimate, the frequentist does not take into account prior beliefs about the likelihood of an event when there are observations available to support computation of its empirical likelihood. The frequentist should take this background information into account when deciding on the threshold for action or conclusion.

As Dikran Marsupial wrote in a concise comment below, "The valuable point the cartoon (perhaps unintentionally) makes is that science is indeed more complex and we can't just apply the "null ritual" without thinking about prior knowledge."

As another example, when trying to determine/declare what topics are "trending" on Facebook, frequentists would likely welcome the more purely algorithmic counting approach Facebook is shifting towards, instead of the old model where employees would curate that list based in part on their own background perspectives about which topics they thought "should" be most important.

$\endgroup$
  • 10
    $\begingroup$ -1. I love XKCD but without any comments I am afraid this is only going to be misleading in this thread. And one needs quite a bit of comments to explain correctly what are the issues pertinent to this comic. If you provide appropriate discussion, I will remove the downvote. $\endgroup$ – amoeba Aug 29 '16 at 23:29
  • 4
    $\begingroup$ It isn't true that the frequentist approach doesn't have the pre-concieved notions, they are encoded in the threshold for statistical significance. ISTR there is a good quote from Fisher about this depending on the nature of the experiment. This is entirely the problem with the frequentist in the cartoon, they just haven't set an appropriate value for $\alpha$ as they haven't taken into account "prior knowledge" about the problem (that the sun going nova is extremely unlikely) and adjusted the test accordingly. Using 95% all the time is part of the much-criticised "null ritual". $\endgroup$ – Dikran Marsupial Aug 30 '16 at 12:17
  • 4
    $\begingroup$ The "null ritual" is bad application of frequentist statistics, but it isn't a problem with frequintism per se. $\endgroup$ – Dikran Marsupial Aug 30 '16 at 12:19
  • 2
    $\begingroup$ @Dikran Your comments seem (incorrectly) to equate "Frequentist" with "user of NHSTs." That is the kind of confusion I have been concerned about. $\endgroup$ – whuber Aug 30 '16 at 13:31
  • 3
    $\begingroup$ @luchonacho FWIW there is a discussion of the XKCD cartoon here stats.stackexchange.com/questions/43339/… The valuable point the cartoon (perhaps unintentionally) makes is that science is indeed more complex and we can't just apply the "null ritual" without thinking about prior knowledge. $\endgroup$ – Dikran Marsupial Aug 30 '16 at 15:37
3
$\begingroup$

(A remark, only tangentially relevant for the question and the site.)

Probability is about objective status of individual things. Things cannot have intention and they receive their statuses from the universe. With a thing, an event (giving it its status) always shall have happened: the event is already there accomplished, even if it haven't actually happened yet - the past future of a thing, also called "fate" or contingency.

Again, with probability, the fact of event - having yet occured or not, doesn't matter - is already there [as opposed to the meaning which never is there]; and as such it's already got unnecessary and superfluous. The fact should be discarded, and that invalidation of it is what we call "the event is probable". Any fact about a thing bears in itself its primeval unconvincing side, or probability of the fact (even the actually occured fact - we recognize it by pinprick of disbelief). We are inevitably "tired of things" pre-psychically to an extent. It remains therefore only to quantify that partial negation of facticity, if need a number. One way to quantify is to count. Another is to weigh. A frequentist carries out or imagines of a series of trials lying before him which he turns face over to see if the event actually happens; he counts. A Bayesian consideres a series of psychological motives dragging behing him which he screens; he weighs them as things. Both men are busy with charge/excuse game of mind. Fundamentally, there is no much difference between them.

Possibility is about potentialities of me in world. Possibility is always mine (a rain's chance is my problem to opt for taking an umbrella or getting wet) and concerns not an object (the one I'm considering as being possible or having the possibility) but the whole world for me. Possibility is always 50/50 and it is always convincing, because it implies - either calls for before or entails after - my decision how to behave. Things themselves have no intentions and thus possibilities. We should not confuse our possibilities of these things for us with their own probabilities of "stochastic determinism". Probability can never be "subjective" in the human sense.


An observant reader may feel in the response a masked dig at the bright answer in this thread, where @amoeba says he thinks "there are almost no frequentists of the [probability definition] kind (P-frequentists)". It could be turned opposite: bayesian probability definers do not exist as different class. Because, as I've admitted, bayesians consider chanks of reality the same manner as frequentists do - as series of facts; only these facts are not experiments, sooner recollections of "truths" and "arguments". But such forms of knowledge is factual and can only be counted or weighed. Probability it erects is not synthesized as subjective, that is, anticipatory ("bayesian" to be) unless human expectation (possibility) enters the scene to meddle. And @amoeba anxiously lets it in when imagines as "the coin will wear off and the Sun will go supernova".

$\endgroup$
0
$\begingroup$

Oh, I've been a frequentist for many's the year,
And I've spent all my time playing the data by ear,
But now I'm returning with Bayes in great store,
And I never will play the frequentist no more.

For it's no nay never, no nay never, no more,
Will I play the frequentist, no never, no more!

I went into a lab where I used to consult.
The gave me some data, said 'p that for us',
I said 'No way, Jose' with a bit of a smile,
P values and evident just don't reconcile!

Chorus

I said it's your prior that we need to shed light,
And the researcher's eyes opened wide with delight,
He said, 'My prior views are as good as the rest,
And for sure a Bayes factor is what will work best!'

Chorus

I'll go back to my teachers, confess what I've done,
And ask them to pardon their prodigal son,
But when they've forgave me, as often before,
I never will play the frequentist no more!

Chorus

And it's no, nay never, no nay never no more,
Will I play the frequentist, no never, no more!

Source: A E Raftery, in The Bayesian Songbook, edited by B P Carlin, at http://www.biostat.umn.edu/. Sung to the traditional folk tune of 'The Wild Rover'. Quoted in Open University M347 Mathematical Statistics, Unit 9.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.