20
$\begingroup$

I've heard the notion that Jaynes claims frequentists operate with an "implicit prior".

What is or are these implicit priors? Does this mean frequentist models are all special cases of Bayesian models waiting to be found?

$\endgroup$
  • $\begingroup$ The implicit prior is a degenerate distribution that puts the entire probability mass at $\theta$, the parameter that the Bayesian frequentist is trying ti estimate. $\endgroup$ – Dilip Sarwate Nov 25 '16 at 3:40
  • 3
    $\begingroup$ As far as I know, there is no frequentist or bayesian model, there are just models and different approaches to them. $\endgroup$ – Andrey Kolyadin Nov 25 '16 at 8:55
  • 3
    $\begingroup$ @DilipSarwate: I disagree with this statement. Using a Dirac mass as a prior does not induce frequentist procedures. And the Bayesian paradigm does not allow for priors with unknown parameters, except when setting another prior on those parameters. $\endgroup$ – Xi'an Nov 25 '16 at 9:19
  • 2
    $\begingroup$ There is always a prior no matter what. Unfortunately all statistical procedures require an ad-hoc starting point which makes them very arbitrary. The good thing is given enough data and correct methodology you get close to your destination. The bad thing is how far you end up from the destination depends on where you start and how much data you have at hand. $\endgroup$ – Cagdas Ozgenc Nov 25 '16 at 12:07
  • 2
    $\begingroup$ @Cagdas Ozgenc: No, there are always assumptions, but they need not take the form of prior distributions. $\endgroup$ – kjetil b halvorsen Nov 25 '16 at 18:33
17
$\begingroup$

In frequentist decision theory, there exist complete class results that characterise admissible procedures as Bayes procedures or as limits of Bayes procedures. For instance, Stein necessary and sufficient condition (Stein. 1955; Farrell, 1968b) states that, under the following assumptions

  1. the sampling density $f(x|\theta)$ is continuous in $\theta$ and strictly positive on $\Theta$; and
  2. the loss function $L$ is strictly convex, continuous and, if $E\subset\Theta$ is compact, $$ \lim_{\|\delta\|\rightarrow +\infty} \inf_{\theta\in E}L(\theta,\delta) =+\infty. $$

an estimator $\delta$ is admissible if, and only if, there exist

  • a sequence $(F_n)$ of increasing compact sets such that $\Theta=\bigcup_n F_n$,
  • a sequence $(\pi_n)$ of finite measures with support $F_n$, and
  • a sequence $(\delta_n)$ of Bayes estimators associated with $\pi_n$ such that

    1. there exists a compact set $E_0\subset \Theta$ such that $\inf_n \pi_n(E_0) \ge 1$
    2. if $E\subset \Theta$ is compact, $\sup_n \pi_n(E) <+\infty$
    3. $\lim_n r(\pi_n,\delta)-r(\pi_n) = 0$ and
    4. $\lim_n R(\theta,\delta_n)= R(\theta,\delta)$.

[reproduced from my book, Bayesian Choice, Theorem 8.3.0, p.407]

In this restricted sense, the frequentist property of admissibility is endowed with a Bayesian background, hence associating an implicit prior (or sequence thereof) with each admissible estimator.

Sidenote: In a sad coincidence, Charles Stein passed away on November 25 in Palo Alto, California. He was 96.

There is a similar (if mathematically involved) result for invariant or equivariant estimation, namely that the the best equivariant estimator is a Bayes estimator for every transitive group acting on a statistical model, associated with the right Haar measure , $\pi^*$, induced on $\Theta$ by this group and the corresponding invariant loss. See Pitman (1939), Stein (1964), or Zidek (1969) for the involved details. This is most likely what Jaynes had in mind, as he argued forcibly about the resolution of the marginalisation paradoxes by invariance principles.

Furthermore, as detailed in civilstat answer, another frequentist notion of optimality, namely minimaxity, is also connected to Bayesian procedures in that the minimax procedure that minimises the maximal error (over the parameter space) is often the maximin procedure that maximises the minimal error (over all prior distributions), hence is a Bayes or limit of Bayes procedure(s).

Q.: Is there a pithy takeaway I can use to transfer my Bayesian intuition to frequentist models?

First I would avoid using the term "frequentist model" as there are sampling models (the data $x$ is a realisation of $X\sim f(x|\theta)$ for a parameter value $\theta$) and frequentist procedures (best unbiased estimator, minimum variance confidence interval, &tc.) Second, I do not see a compelling methodological or theoretical reason for considering frequentist methods as borderline or limiting Bayesian methods. The justification to a frequentist procedure, when it exists, is to satisfy some optimality property in the sampling space, that is when repeating the observations. The primary justification for Bayesian procedures is to be optimal [under a specific criterion or loss function] given a prior distribution and one realisation from the sampling model. Sometimes, the resulting procedure satisfies some frequentist property (the $95$% credible region is a $95$% confidence region), but this is happenstance in that this optimality does not transfer to all procedures associated with the Bayesian model.

$\endgroup$
  • 1
    $\begingroup$ Thanks very much. As a novice, Is there a pithy takeaway I can use to transfer my bayesian intuition to frequentist models? ie (this GLM is similiar to x with y prior, or this lasso is like bayesian xyz). $\endgroup$ – Bayesquest Nov 25 '16 at 14:55
  • 1
    $\begingroup$ Also, would you mind taking a look at my other question here: stats.stackexchange.com/questions/247850/… I know you have proposed some solutions to the bayesian brittleness issue...but I have a feeling the solutions are not robust or easy to wield for a social scientist. $\endgroup$ – Bayesquest Nov 25 '16 at 16:24
  • 1
    $\begingroup$ For the first comment, here are some examples of what I was talking about: - Neural nets & GPs - stats.stackexchange.com/questions/71782/… - sumsar.net/blog/2015/04/… - [A nonparametric Bayesian (npB) pointof-view allows interpretation of forests as a sample from a posterior over trees] (arxiv.org/pdf/1502.02312.pdf) $\endgroup$ – Bayesquest Nov 25 '16 at 18:28
  • $\begingroup$ We worked on Approximate Bayesian Inference with random forests and found that the variability resulting from that tool was rather unrelated with the original posterior. Of course, this does not mean it does not allow for a Bayesian interpretation but nonetheless... $\endgroup$ – Xi'an Nov 26 '16 at 14:07
12
$\begingroup$

@Xi'an's answer is more complete. But since you also asked for a pithy take-away, here's one. (The concepts I mention are not exactly the same as the admissibility setting above.)

Frequentists often (but not always) like to use estimators that are "minimax": if I want to estimate $\theta$, my estimator $\hat{\theta}$'s worst-case risk should be better than any other estimator's worst-case risk. It turns out that MLEs are often (approximately) minimax. See details e.g here or here.

In order to find the minimax estimator for a problem, one way is to think Bayesian for a moment and find the "least favorable prior" $\pi$. This is the prior whose Bayes estimator has higher average risk than any other prior's Bayes estimator. If you can find it, then it turns out $\pi$'s Bayes estimator is minimax.

In this sense, you could pithily say: A (minimax-using) Frequentist is like a Bayesian who chose (the point estimate based on) a least-favorable prior.

Maybe you could stretch this to say: Such a Frequentist is a conservative Bayesian, choosing not subjective priors or even uninformative priors but (in this specific sense) worst-case priors.

Finally, as others have said, it's a stretch to compare Frequentists and Bayesians in this way. Being a Frequentist doesn't necessarily imply that you use a certain estimator. It just means that you ask questions about your estimator's sampling properties, whereas these questions are not a Bayesian's top priority. (So any Bayesian who hopes for good sampling properties, e.g. "calibrated Bayes," is also a Frequentist.)
Even if you define a Frequentist as one whose estimators always have optimal sampling properties, there are many such properties and you can't always meet them all at once. So it's hard to speak generally about "all Frequentist models."

$\endgroup$
  • 4
    $\begingroup$ I thought that an implicit prior for frequentist analysis would be some uniform prior. $\endgroup$ – Michael R. Chernick Nov 26 '16 at 11:40
  • 5
    $\begingroup$ It can be, sometimes. You could think of an MLE as the MAP estimate using a uniform prior. But MLEs are not the only tool Frequentists use. $\endgroup$ – civilstat Nov 26 '16 at 14:36
  • $\begingroup$ Another related concept: "matching priors" or "probability matching priors," specific priors designed s.t. your $1-\alpha$ credible interval approximately matches the frequentist $1-\alpha$ confidence interval for that particular parameter. Again, these can be uniform but don't have to be. Depends on the choice of parameter and on how good you want the approximation to be. See for instance utstat.utoronto.ca/reid/research/vaneeden.pdf $\endgroup$ – civilstat Dec 14 '16 at 2:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.