# Introduction to frequentist statistics for Bayesians [closed]

I'm a simple minded Bayesian who feels comfortable in the cosy world of Bayes.

However, due to malevolent forces outside my control, I now have to do introductory graduate courses about the exotic and weird world of frequentist statistics. Some of these concepts seem very weird to me, and my teachers are not versed in Bayes, so I thought I'd get some help on the internet from those who understand both.

How would you explain the different concepts in frequentist statistics to a Bayesian who finds frequentism weird and uncomfortable?

For example, some things I already understand:

• The maximum likelihood estimator $\text{argmax}_\theta \;p(D|\theta)$ is equal to the maximum posterior estimator $\text{argmax}_\theta \;p(\theta |D)$, if $p(\theta)$ is flat.
• (not entirely sure about this one). If a certain estimator $\hat \theta$ is a sufficient statistic for a parameter $\theta$, and $p(\theta)$ is flat, then $p(\hat \theta|\theta)=c_1\cdot p(D|\theta)=c_1\cdot c_2\cdot p(\theta|D)$, i.e. the sampling distribution is equal to the likelihood function, and therefore equal to the posterior of the parameter given a flat prior.

Those are examples of explaining frequentist concepts to someone who understands Bayesian ones.

How would you similarly explain the other central concepts of frequentist statistics in terms a Bayesian can understand?

## Specifically, I'm interested in the following questions:

• What is the role of Mean Square Error? How does it relate to Bayesian loss functions?
• How does the criterion of "unbiasedness" relate to Bayesian criteria? I know that a Bayesian will not demand that its estimators are unbiased, but at the same time, a Bayesian would probably agree that an unbiased frequentist estimator is generally more desirable than a biased frequentist one (even though he would consider both to be inferior to the Bayesian estimator). So how does a Bayesian understand unbiasedness?
• If we have flat priors, do frequentist confidence intervals somehow coincide with Bayesian ones?
• What in the name of Laplace is going on with specification tests like the $F$ test? Is this some degenerate special case of a Bayesian update on the distribution over model space?

## More generally:

Is there some resource that explains frequentism to Bayesians? Most of the books run the other way around: they explain Bayesianism to people who are experienced in frequentist statistics.

ps. I have looked, and while there are a lot of questions already about the difference between Bayesian and Frequentism, none explicitly explain Frequentism from the perspective of a Bayesian.

This question is related, but is not specifically about explaining Frequentist concepts to a Bayesian (more about justifying frequentist thinking in general).

Also, my point is not to bash frequentism. I really do want to understand it better

• This may be a valid question but it sounds too broad at this form, since you seem to ask us to translate any possible frequentist method into Bayesian form -- one could write a book on this topic! Most introductory Bayesian handbooks (e.g. Kruschke) discuss those topics as they are aimed at frequentists. Moreover, you probably already use a lot of frequentist tools (e.g. for diagnosing your MCMC chains). Some of the topics you mention are not purely Bayesian (loss function). Could you try editing your question to make it more specific? – Tim Nov 14 '17 at 8:27
• I sympathise and appreciate the question. In addition to literature, if you are interested in insights from the community on specific questions I recommend you try to split. Many of them sound very interesting. I have to admit I have asked myself several of these questions with mixed success. – Benoit Sanchez Nov 14 '17 at 9:15
• This question made me cringe and think Statistics education is broken. How someone managed to go through basic Statistics without the notion of bias is baffling. Similarly the fact that a student finds his educators not "versed in Bayes" so they cannot explain "Frequentist Stats" adequately to a "Bayesian" is I don't know... awkward? (I don't downvote or anything, I think it is good that the OP asks and TIm's answer below is very sensible (+1) but really... the question made me uneasy about something being fundamentally wrong in this context.) – usεr11852 says Reinstate Monic Nov 14 '17 at 9:21
• You are not going to really understand rock climbing if you force yourself to see it like vertical swimming. – David Ernst Nov 15 '17 at 5:53
• Take a simple t-test then, it doesn't compute a posterior because it explicitly doesn't want to choose a prior. How are you going to find a Bayesian equivalent for avoiding Bayes? With MLE you have found the one technique where you will easily identify a Bayesian equivalent since this is basically already a Bayesian technique in all but name. – David Ernst Nov 15 '17 at 13:56

Actually many of the things mentioned by you are already discussed by the major Bayesian handbooks. In many cases those handbooks are written for frequentists by training, so they discuss many similarities and try translating the frequentist methods into Bayesian ground. One example is the Doing Bayesian Data Analysis book by John K. Kruschke or his paper translating $t$-test into Bayesian ground. There is also another psychologist, Eric-Jan Wagenmakers who with his team talked a lot about translating frequentist concepts into Bayesian ground. Decision-theoretic concepts like loss functions, unbiassness etc. are discussed in the The Bayesian Choice book by Christian P. Robert.

Moreover, some of the concepts mentioned by you are not really Bayesian. For example, loss function is a general concept and only if you combine it with prior distribution you get a Bayes risk.

It is also worth mentioning that even if you are self-declared Bayesian, then you probably already use a lot of frequentist methods. For example, if you use MCMC for estimation and then calculate mean of the MCMC chain as your point estimate, then you are using a frequentist estimator, since you are not using any Bayesian model and priors to get the estimate of the mean of the MCMC chain.

Finally, some frequentist concepts and tools are not easily translatable to Bayesian setting, or the proposed "equivalents" are rather proofs of concept, then something that you'd use in real life. In many cases the approaches are simply different and looking for parallels is a waste of time.

• (+1): very good points! – Xi'an Nov 14 '17 at 12:19
• You could find this blog interesting: errorstatistics.com – kjetil b halvorsen Nov 14 '17 at 15:30

(not entirely sure about this one). If a certain estimator $\hat θ̂$ is a sufficient statistic for a parameter $θ$, and $p(θ)$ is flat, then $p(\hat θ̂ |θ)=p(D|θ)=c⋅p(θ|D)$, i.e. the sampling distribution is equal to the likelihood function, and therefore equal to the posterior of the parameter given a flat prior.

This is incorrect:

1. $p(D|θ)=p(\hat θ̂ |θ)\times p(D|\hat θ)$ when $\hat θ$ is a sufficient statistic
2. $p(D|θ)=c⋅p(θ|D)$ is false when considered as a function of $D$, and when considered as a function of $θ$ (unless one uses the flat prior)
3. only does the posterior based on $\hat θ$ equal the posterior based on $D$ in this context.

Furthermore, sufficiency has nothing to do with frequentism versus Bayesianism, even though there exist specifically Bayesian notions of sufficiency. As for instance in model comparison.

a Bayesian would probably agree that an unbiased frequentist estimator is generally more desirable than a biased frequentist one

The trouble with this part of the question is that Bayesian estimators are frequentist estimators as well in that they satisfy frequentist properties like admissibility or sometimes minimaxity. As discussed in a recent CV entry, Bayes estimates under squared error loss cannot be unbiased. And there is no reason beyond using a special loss function to favour unbiasedness: minimising a posterior loss is all-inclusive and if imposing unbiasedness results in a higher loss it should not be considered. (A last point is that there are very few functions of the parameter that allow for unbiased estimators.)

• Just to clarify, that point was not an attempt to connect the concept of sufficiency to Bayesianism, but to connect the concept of sampling distributions to posterior distributions. The concepts of sufficiency in frequentism and bayesianism are equivalent, which can be seen by simply applying bayes' rule. But I will study your post and try to understand this better. – user56834 Nov 14 '17 at 12:31

It appears to me as if you are considering a world of frequentists and Bayesians. That is not much nuanced. Like if you have to be the one or the other, or as if the methods applied are determined by some personal believes (rather than convenience and the specific problem and information at hand). I believe that this is a misconception based on current trends in calling oneself a frequentist or Bayesian, and also lots of statistical language may be confusing. Just try to have a group of statisticians explain p-value or confidence interval.

Some classical works may help you to understand frequentist inference. The classical works contain fundamental principles, are close to the heat of the discussion between proponents, and provide a background of the (practical) motivation and relevance at that time.

also, these classical works on frequentist methods, were written in a time when people mostly worked with Bayesian principles and mathematical calculation of probability (note that statistics is not always as if you are working on a typical mathematics problem with probabilities, the probabilities may be very ill-defined).

# Frequentist probability is not inverse probability

'Inverse probability' Fisher 1930

You make a notion of the likelihood as being a Bayesian expression with a flat prior

However,

1. while the mathematics coincide (when wrongly interpreted, since you may get P(x|a) = P(a|x), up to a constant, but they are not the same terms) the construction and meaning is different.

2. Likelihood is not meant to be a 'Bayesian probability based on flat, or uniformed, priors'. Likelihood is not even a probability and does not follow the rules of probability distributions (for instance you can not add up likelihood for different events, and the integral is not equal to one), it is only when you multiply it with a flat prior, that it becomes a probability, but then the meaning has changed as well.

Some interesting quotes from 'inverse probability' 1930 Fisher.

Bayesian and frequentist methods are different tools:

...there are two different measures of rational belief appropriate to different cases. Knowing the population we can express our incomplete knowledge of, or expectation of, the sample in terms of probability; knowing the sample we can express our incomplete knowledge of the population in terms of likelihood. We can state the relative likelihood that an unknown correlation is + 0.6, but not the probability that it lies in the range .595-.605.

Note that there is a certain probability statement, which a frequentist method provides.

By constructing a table of corresponding values, we may know as soon as T is calculated what is the fiducial 5 per cent, value of $\theta$, and that the true value of $\theta$ will be less than this value in just 5 per cent, of trials. This then is a definite probability statement about the unknown parameter $\theta$, which is true irrespective of any assumption as to its a priori distribution.

• a frequentist method makes a statement about the probability that an experiment (with random interval) will have the true value of a (possibly random) parameter inside the interval given by a statistic.
• This is not the be confused with the probability that a specific experiment (with fixed interval) will have the true value of the (fixed) parameter inside the interval given by the statistic.

See also 'On the "Probable Error" of a Coefficient of Correlation Deduced from a Small Sample.' Fisher 1921 in which Fisher demonstrated the difference of his method not being a Bayesian inverse probability.

In the former paper it was found, by applying a method previously developed, that the << most likely >> value of the correlation of the population was, numerically, slightly smaller than that of the sample. This conclusion was adversely criticized in Biometrica, apparently on the incorrect assumption that I had deduced it from Bayes theorem. It will be shown in this paper that when the sampling curves are rendered approximately normal, the correction I had proposed is equal to the distance between the population value and the mid-point of the sampling curve and is accordingly no more than the correction of a constant bias introduced by the method of calculation. No assumption as to a priori probability is involved.

and

...two radically distinct concepts have been confused under the name of << probability >> ...

that is probability and likelihood. See also the note on the end of Fishers article from 1921 in which he speaks more on the confusion.

Note again that likelihood is a function of a set of parameters, but not a probability density function of that set of parameters.

Probability is used for something you can observe. E.g the probability that a dice rolls six. Likelihood is used for something that you can not observe, e.g. the hypothesis that a dice rolls six 1/6 of the time.

also, you might like Fisher's work in which he is much lighter in his opinion on Bayes theorem (still describing the differences). 'On the mathematical foundations of theoretical statistics' Fisher 1922 (especially section 6 'formal solution of problem of estimation')

# More

If you can understand and appreciate the comments from Fisher on the difference between inverse probability and the principle of likelihood you may wish to read further on differences within frequentist methods.

'Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability' Neyman 1937

Which is a work of 50 pages and difficult to summarize. But it deals with your questions on unbiasedness, explains the method of least squares (and difference with method of maximum likelihood), and specifically provides a treatment of confidence intervals (frequentist interval are already not similar, unique, let alone that the are the same as Bayesian intervals for flat priors).

Regarding the F-test it is not clear, what in the name of Laplace you think is wrong. If you like an early use you can look in 'Studies in crop variation. II. The manurial response of different potato varieties' 1923 Fisher and Mackenzie

This paper has the expression of anova in a recognizable linear model subdividing the sums of squares into between and within groups.

(in the test of the 1923 article the test consists of a comparison of differences between the logs of sample standard deviations with a calculated standard error for this difference that is determined by a sum of degrees of freedom $\frac{1}{2d_1} + \frac{1}{2d_2}$. Later works make this more sophisticated expressions leading to the F-distribution, such that it may diffuse the ideas that one may have about it. But in essence, without the technical juggling due to more exact distributions for small numbers, it's origin is much like a z-test).