13
$\begingroup$

Bayesian statistics is criticized for being subjective, as it requires a prior distribution encapsulating the subjective befiefs of the observer. Frequentist statistics is commonly advertised as being objective (because it does not require choosing a prior, because its probabilities are defined in an objective way, or whatever reason I'm not really sure about), however Wald proved that most admissible decision rules in Frequentist statistics are each one equivalent to a Bayes rule using some prior (which may be improper among other things, but it exists). One could think of the frequentist rules as being uninformative, but the calculation of uninformative priors may yield results different to the frequentist ones (as in the case of the Bernoulli distribution, for example). My questions are the following:

  1. Is there an objective reason to choose the priors associated to frequentist decision rules over others?
  2. If not, does that mean that frequentist inference is subjective?
  3. If yes, what is the personal belief / knowledge that we are injecting when using frequentist inference?

Thanks in advance!

$\endgroup$
4
  • 1
    $\begingroup$ I suppose you are interested in situations where the likelihood is known a priori? In real world problems, the likelihood is also a subjective choice, which makes both paradigms obviously subjective. $\endgroup$ – JTH Sep 22 '20 at 18:25
  • 2
    $\begingroup$ @JTH Not quite. The shape of the likelihood function is an aspect of the "theoretical model" or "hypothesis", and like all other aspects of the theoretical model, it can be tested (and compete against other theoretical models with different likelihood shapes) for consistency with data. $\endgroup$ – Daniel Hatton Sep 22 '20 at 19:15
  • $\begingroup$ @Xi'an that's precisely why I ask: I sympathize with the idea that's all priors are somehow informative, so if frequentist decision rules are equvalent to some priors, the logical conclusion is that frequentist inference is informative, and thus subjective. How could one argue against this conclusion? that why I posted this question. $\endgroup$ – Daniel Turizo Sep 22 '20 at 19:25
  • 1
    $\begingroup$ Whenever you are in doubt choose this subjective rule: $\theta = 42$ (and whatever decision that may follow from it). That rule is gonna be admissible too. Choosing a different admissible rule, like a frequentist estimate, is gonna be a subjective choice (a subjective choice to be objective). $\endgroup$ – Sextus Empiricus Sep 23 '20 at 14:26
7
$\begingroup$

Let me recall (from my book) the precise setting of Wald's characterisation of admissible estimators: first, Stein's theorems for admissible procedures to be limit of Bayes procedures:

Charles Stein (1955) produced a necessary and sufficient condition: if (i) $f(x|\theta)$ is continuous in $\theta$ and strictly positive on $\Theta$; and (ii) the loss function $\text{L}(\cdot,\cdot)$ is strictly convex, continuous and, if $E\subset\Theta$ is compact, $$ \lim_{\|\delta\|\rightarrow +\infty} \inf_{\theta\in E} \text{L}(\theta,\delta) =+\infty, $$ then an estimator $\delta$ is admissible if, and only if, there exist (a) a sequence $(F_n)$ of increasing compact sets such that $\Theta=\bigcup_n F_n$, (b) a sequence $(\pi_n)$ of finite measures with support $F_n$, and (c) a sequence $(\delta_n)$ of Bayes estimators associated with $\pi_n$ such that

  1. there exists a compact set $E_0\subset \Theta$ such that $\inf_n \pi_n(E_0) \ge 1$;
  2. if $E\subset \Theta$ is compact, $\sup_n \pi_n(E) <+\infty$;
  3. $\lim_n r(\pi_n,\delta)-r(\pi_n) = 0$; and
  4. $\lim_n R(\theta,\delta_n)= R(\theta,\delta)$.

Larry Brown (1986) provides an alternative, and quite general, characterization of admissible estimators. Consider $x\sim f(x|\theta)$, and assume $\text{L}$ to be lower semi-continuous and such that $$ \lim_{||\delta||\rightarrow +\infty} \text{L}(\theta,\delta) = +\infty. $$ Brown (1986) shows that, under these conditions, the closure (for the pointwise convergence) of the set of all Bayes estimators is a complete class.

Proposition If L is strictly convex, every admissible estimator of $\theta$ is a pointwise limit of Bayes estimators for a sequence of priors with finite supports.

Second, the generic Wald (1950)'s complete class result:

Theorem Consider the case when $\Theta$ is compact and the risk set $$ \mathcal R = \{(R(\theta,\delta))_{\theta\in\Theta},\ \delta\in\mathcal D^*\}, $$ is convex (where $\mathcal D^*$ denotes the set of randomised decisions). If all estimators have a continuous risk function, the Bayes estimators constitute a complete class.

and a remark about cases when it does not hold:

In the case of distributions with discrete support, the completeness of generalised Bayes estimators does not always hold and complete classes involve piecewise-Bayesian procedures (see Berger and Srinivasan (1978), Brown (1981), and Brown and Farrell (1985)).

These results do not imply that every admissible estimator can be associated with a proper prior or an improper prior. Furthermore, even if this is the case, there are as many "admissible" priors as there are admissible estimators, hence no apparent restriction on the choice of priors. (This is why admissibility is a desirable feature rather than an optimality property per se.)

In the same way that the notion of an "objective", "uninformative", "default" prior does not meet a consensus in the Bayesian community, there is no consensus about a default frequentist procedure that would lead to the notion of a "frequentist prior". Note in addition that associating a prior with a frequentist procedure is depending on the choice of the loss function L, hence varying with the quantity of interest.

$\endgroup$
2
  • $\begingroup$ So frequentists choose their decision rules (estimators) based on some properties that are desirable under their set of beliefs (intuitively "fair" as Efron said in Why isn't Everyone a Bayesian) like unbiasedness, MSE optimal, etc; objective bayesians choose their decision rules (priors) based on some properties that are desirable under their set of beliefs, like maximum-entropy. In the end both are subjective in the sense that you have to accept a set of beliefs, and both are objective in the sense that under their set of beliefs the decisions are sensible or "fair". Is this correct? $\endgroup$ – Daniel Turizo Sep 23 '20 at 15:58
  • $\begingroup$ @SextusEmpiricus: Note that the theorem I provide assume $\Theta$ is compact, in which case the sample mean is not admissible under square error. The extension to non-compact settings with complete class results encompasses improper priors as well. This includes the sample mean (although it is not admissible for dimensions $p\ge 3$). $\endgroup$ – Xi'an Sep 23 '20 at 16:29
2
$\begingroup$

I discussed your question 2 at some length in appendix E of my Ph.D. thesis (Hatton, 2003, Spin-polarized electron scattering at ferromagnetic interfaces, University of Cambridge). The position I eventually reached was that true objectivity is achieved when (usually due to having lots of data) key features of the posterior distribution become independent of the choice of prior over some domain of "reasonable" priors. The frequentist approach, viewed as having a prior in the way you suggest, conceals the prior and renders it immutable, which gets in the way of testing for objectivity, when objectivity is conceived in the way I suggested.

BTW, I didn't know about Wald's proof that 'every admissible decision rule in Frequentist statistics is equivalent to a Bayes rule using an appropiately chosen prior'. Do you have a specific citation? (In my thesis (appendix D), I gave my own proof, but mine only works for a certain subset of frequentist significance tests.)

$\endgroup$
12
  • 1
    $\begingroup$ Regarding the reference, Murphy's book Machine Learning: A Probabilistic Perspective states the theorem (Theorem 6.3.2). $\endgroup$ – Daniel Turizo Sep 22 '20 at 19:03
  • 1
    $\begingroup$ @DanielTurizo Well, one can assert a priori anything one likes; that's what "a priori" means. But I personally wouldn't assert that the frequentist prior is objective; nor, I think, would anyone who even slightly identifies as a Bayesian. Thanks for the reference. $\endgroup$ – Daniel Hatton Sep 22 '20 at 19:12
  • 2
    $\begingroup$ I do not understand the discussion of a "frequentist prior". For instance, there is an infinity of admissible estimators, each of which can be associated with a prior or a limit of priors when Wald's complete class applies. Which one is the "frequentist prior"? Which one is "objective"? $\endgroup$ – Xi'an Sep 23 '20 at 6:36
  • 1
    $\begingroup$ @DanielHatton Test statistics can be seen as estimators, so you could think of it like that (btw I couldn't find your thesis, would you share?) $\endgroup$ – Daniel Turizo Sep 23 '20 at 16:01
  • 2
    $\begingroup$ @DanielTurizo 'I couldn't find your thesis, would you share' While I still have the full source code of my thesis, it currently won't compile into human-readable form, primarily because Gnuplot's command language has changed in some non-backwards-compatible ways since 2003. Last night, I wrote a Perl script to update the Gnuplot scripts to the new syntax, but I haven't captured all the changes yet. If I can make it work reasonably straightforwardly, I'll put my thesis on GitHub under the GNU FDL. $\endgroup$ – Daniel Hatton Sep 23 '20 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.