What is/are the implicit priors in frequentist statistics?

Question

I've heard the notion that Jaynes claims frequentists operate with an "implicit prior".

What is or are these implicit priors? Does this mean frequentist models are all special cases of Bayesian models waiting to be found?

Answer 1

In frequentist decision theory, there exist complete class results that characterise admissible procedures as Bayes procedures or as limits of Bayes procedures. For instance, Stein necessary and sufficient condition (Stein. 1955; Farrell, 1968b) states that, under the following assumptions

1. the sampling density $f(x|\theta)$ is continuous in $\theta$ and strictly positive on $\Theta$; and
2. the loss function $L$ is strictly convex, continuous and, if $E\subset\Theta$ is compact, $$ \lim_{\|\delta\|\rightarrow +\infty} \inf_{\theta\in E}L(\theta,\delta) =+\infty. $$

an estimator $\delta$ is admissible if, and only if, there exist

• a sequence $(F_n)$ of increasing compact sets such that $\Theta=\bigcup_n F_n$,
• a sequence $(\pi_n)$ of finite measures with support $F_n$, and

• a sequence $(\delta_n)$ of Bayes estimators associated with $\pi_n$ such that

  1. there exists a compact set $E_0\subset \Theta$ such that $\inf_n \pi_n(E_0) \ge 1$
  2. if $E\subset \Theta$ is compact, $\sup_n \pi_n(E) <+\infty$
  3. $\lim_n r(\pi_n,\delta)-r(\pi_n) = 0$ and
  4. $\lim_n R(\theta,\delta_n)= R(\theta,\delta)$.

[reproduced from my book, Bayesian Choice, Theorem 8.3.0, p.407]

In this restricted sense, the frequentist property of admissibility is endowed with a Bayesian background, hence associating an implicit prior (or sequence thereof) with each admissible estimator.

Sidenote: In a sad coincidence, Charles Stein passed away on November 25 in Palo Alto, California. He was 96.

There is a similar (if mathematically involved) result for invariant or equivariant estimation, namely that the the best equivariant estimator is a Bayes estimator for every transitive group acting on a statistical model, associated with the right Haar measure , $\pi^*$, induced on $\Theta$ by this group and the corresponding invariant loss. See Pitman (1939), Stein (1964), or Zidek (1969) for the involved details. This is most likely what Jaynes had in mind, as he argued forcibly about the resolution of the marginalisation paradoxes by invariance principles.

Furthermore, as detailed in civilstat answer, another frequentist notion of optimality, namely minimaxity, is also connected to Bayesian procedures in that the minimax procedure that minimises the maximal error (over the parameter space) is often the maximin procedure that maximises the minimal error (over all prior distributions), hence is a Bayes or limit of Bayes procedure(s).

Q.: Is there a pithy takeaway I can use to transfer my Bayesian intuition to frequentist models?

First I would avoid using the term "frequentist model" as there are sampling models (the data $x$ is a realisation of $X\sim f(x|\theta)$ for a parameter value $\theta$) and frequentist procedures (best unbiased estimator, minimum variance confidence interval, &tc.) Second, I do not see a compelling methodological or theoretical reason for considering frequentist methods as borderline or limiting Bayesian methods. The justification to a frequentist procedure, when it exists, is to satisfy some optimality property in the sampling space, that is when repeating the observations. The primary justification for Bayesian procedures is to be optimal [under a specific criterion or loss function] given a prior distribution and one realisation from the sampling model. Sometimes, the resulting procedure satisfies some frequentist property (the $95$% credible region is a $95$% confidence region), but this is happenstance in that this optimality does not transfer to all procedures associated with the Bayesian model.

Answer 2

@Xi'an's answer is more complete. But since you also asked for a pithy take-away, here's one. (The concepts I mention are not exactly the same as the admissibility setting above.)

Frequentists often (but not always) like to use estimators that are "minimax": if I want to estimate $\theta$, my estimator $\hat{\theta}$'s worst-case risk should be better than any other estimator's worst-case risk. It turns out that MLEs are often (approximately) minimax. See details e.g here or here.

In order to find the minimax estimator for a problem, one way is to think Bayesian for a moment and find the "least favorable prior" $\pi$. This is the prior whose Bayes estimator has higher average risk than any other prior's Bayes estimator. If you can find it, then it turns out $\pi$'s Bayes estimator is minimax.

In this sense, you could pithily say: A (minimax-using) Frequentist is like a Bayesian who chose (the point estimate based on) a least-favorable prior.

Maybe you could stretch this to say: Such a Frequentist is a conservative Bayesian, choosing not subjective priors or even uninformative priors but (in this specific sense) worst-case priors.

Finally, as others have said, it's a stretch to compare Frequentists and Bayesians in this way. Being a Frequentist doesn't necessarily imply that you use a certain estimator. It just means that you ask questions about your estimator's sampling properties, whereas these questions are not a Bayesian's top priority. (So any Bayesian who hopes for good sampling properties, e.g. "calibrated Bayes," is also a Frequentist.)
Even if you define a Frequentist as one whose estimators always have optimal sampling properties, there are many such properties and you can't always meet them all at once. So it's hard to speak generally about "all Frequentist models."