15
$\begingroup$

For a given inference problem, we know that a Bayesian approach usually differ in both form and results from a fequentist approach. Frequentists (usually includes me) often point out that their methods don't require a prior and hence are more "data driven" than "judgement driven". Of course, Bayesian's can point to non-informative priors, or, being pragmatic, just use a really diffuse prior.

My concern, especially after feeling a hint of smugness at my fequentist objectivity, is that perhaps my purportedly "objective" methods can be formulated in a Bayesian framework, albeit with some unusual prior and data model. In that case, am I just blissfully ignorant of the preposterous prior and model my frequentist method implies?

If a Bayesian pointed out such a formulation, I think my first reaction would be to say "Well, that's nice that you can do that, but that's not how I think about the problem!". However, who cares how I think about it, or how I formulate it. If my procedure is statistically/mathematically equivalent to some Bayesian model, then I am implicitly (unwittingly!) performing Bayesian inference.

Actual Question Below

This realization substantially undermined any temptation to be smug. However, I'm not sure if its true that the Bayesian paradigm can accommodate all frequentist procedures (again, provided the Bayesian chooses a suitable prior and likelihood). I know the converse is false.

I ask this because I recently posted a question about conditional inference, which led me to the following paper: here (see 3.9.5,3.9.6)

They point out Basu's well-known result that there can be more than one ancillary statistic, begging the question as to which "relevant subset" is most relevant. Even worse, they show two examples of where, even if you have a unique ancillary statistic, it does not eliminate the presence of other relevant subsets.

They go on to conclude that only Bayesian methods (or methods equivalent to them) can avoid this problem, allowing unproblematic conditional inference.


It may not be the case that Bayesian Stats $\supset$ Fequentist Stats -- that's my question to this group here. But it does appear that a fundamental choice between the two paradigms lies less in philosophy than in goals: do you need high conditional accuracy or low unconditional error:

  • High conditional accuracy seems applicable when we have to analyze a singular instance -- we want to be right for THIS particular inference, despite the fact that this method may not be appropriate or accurate for the next dataset (hyper-conditionality/specialization).

  • Low unconditional error is appropriate when if we are willing make conditionally incorrect inferences in some cases, so long as our long run error is minimized or controlled. Honestly, after writing this, I'm not sure why I would want this unless I were strapped for time and couldn't do a Bayesian analysis...hmmm.

I tend to favor likelihood-based fequentist inference, since I get some (asymptotic/approximate) conditionality from the likelihood function, but don't need to fiddle with a prior - however, I've become increasingly comfortable with Bayesian inference, especially if I see the prior a a regularization term for small sample inference.

Sorry for the aside. Any help for my main problem is appreciated.

$\endgroup$
  • 1
    $\begingroup$ an edit to the post seems to have messed up the link, but there isn't enough information for me to fix it. $\endgroup$ – Dikran Marsupial Jul 27 '16 at 10:18
  • $\begingroup$ @DikranMarsupial thanks for noticing that. I added back in the link. $\endgroup$ – user75138 Jul 28 '16 at 14:55
3
$\begingroup$

I would argue that frequentists are indeed often "implicit/unwitting Bayesians", as in practice we often want to perform probabilistic reasoning about things that don't have a long run frequency. The classic example being Null Hypothesis Statistical Testing (NHST), where what we really want to know is the relative probabilities of the Null and Research Hypotheses being true, but we cant do this in a frequentist setting as the truth of a particular hypothesis has no (non-trivial) long run frequency - it is either true or it isn't. Frequentist NHSTs get around this by substituting a different question, "what is the probability of observing an outcome at least as extreme under the null hypothesis" and then compare that to a pre-determined threshold. However this procedure does not logically allow us to conclude anything about whether H0 or H1 is true, and in doing so we are actually stepping out of a frequentist framework into a (usually subjective) Bayesian one, where we conclude that the probability of observing such an extreme value under H0 is so low, that we can no longer believe that H0 is likely to be true (note this is implicitly assigning a probablility to a particular hypothesis).

Note it isn't actually true that frequentist procedures don't have subjectivity or priors, in NHSTs the threshold on the p-value, $\alpha$, serves much the same purpose as the priors $p(H_0)$ and $p(H_1)$ in a Bayesian analysis. This is illustrated by the much-discussed XKCD cartoon:

enter image description here

The main reason the frequentists conclusion is unreasonable is that the value of $\alpha$ does not represent a reasonable state of knowledge regarding the detector and/or solar physics (we know that it is extremely unlikely that the sun has exploded, and rather less so that the detector has a false alarm). Note in this case the conclusion that the sun has exploded inferred from a low p-value (a Bayesian inference) but is not logically entailed by it. The subjectivity is still there, but not stated explicitly in the analysis and often neglected.

Arguably confidence intervals are often used (and interpreted as) an interval in which we can expect to see the observations with a given probability, which again is a Bayesian interpretation.

Ideally statisticians ought to be aware of the benefits and disadvantages of both approaches and be prepared to use the right framework for the application at hand. Basically we should aim to use the analysis that provides the most direct answer to the question we actually want answered (and not quietly substitute a different one), so a frequentist approach is probably most efficient where we actually are interested in long-run frequencies and Bayesian methods where that is not the case.

I suspect that most frequentist questions can be answered by a Bayesian as there is nothing to stop a Bayesian from answering questions like "what is the probability of observing a result at least as extreme if $H_0$ is true", however I'll need to do a bit of reading on that one, interesting question.

$\endgroup$
  • 2
    $\begingroup$ I agree with your analysis of this cartoon. It's funny, but I sometimes see people bringing it up as some argument against frequentist statistics and that's just silly. "Extraordinary claims require extraordinary evidence"; no sane frequentist statistician would be willing to take some meager $p<0.05$ evidence to support such an extraordinary claim as here. $\endgroup$ – amoeba says Reinstate Monica Jul 27 '16 at 22:46
  • 3
    $\begingroup$ I think it works as a criticism of "null ritual" statistics, where NHSTs are used as a recipe from the statistics cookbook without thinking/understanding the procedure (probably including me when I started out ;o). $\endgroup$ – Dikran Marsupial Jul 28 '16 at 6:39
  • 2
    $\begingroup$ I think a big problem with NHST is less the ritualized p<0.05 and more that it artificially restricts the space of hypotheses. I mean, they are not considering (as hypotheses) the numerous other reasons that it would be dark other than "the sun exploded". This is what I really dislike about NHST...it's so conservative towards the null. Also, I wonder what the power of this test would be? $\endgroup$ – user75138 Jul 28 '16 at 15:04
  • 1
    $\begingroup$ IMHO the main value of NHSTs is as a sanity-check hurdle to stop us getting overly excited about our research hypothesis, so favouring the null is perhaps not such a bad thing. The p-value fallacy (interpreting it as the probability that $H_0$ is true) is another major problem (again caused by a it being treated as a direct answer to the question we want answered). p<0.05 is a reasonable sanity check for many practical applications, but not universal. ISTR there is a good quote from Fisher about this depending on the nature of the analysis, but I can never find it when I want it! $\endgroup$ – Dikran Marsupial Jul 28 '16 at 15:19
  • $\begingroup$ @DikranMarsupial agree! Fisher envisioned them assisting with agricultural experiments, which is where the p-value came from. Evaluating pesticides and fertilizers is nothing like evaluating non-replicable existential questions...wrong tool for the job. $\endgroup$ – user75138 Jul 28 '16 at 16:03
4
$\begingroup$

Bayesians and Frequentists do not only differ in how they obtain inferences, or how similar or different these inferences can be uncertain certain prior choices. The main difference is how they interpret probability:

Bayesian probability:

Bayesian probability is one interpretation of the concept of probability. In contrast to interpreting probability as frequency or propensity of some phenomenon, Bayesian probability is a quantity that is assigned to represent a state of knowledge, or a state of belief.

Frequentist probability:

Frequentist probability or frequentism is a standard interpretation of probability; it defines an event's probability as the limit of its relative frequency in a large number of trials. This interpretation supports the statistical needs of experimental scientists and pollsters; probabilities can be found (in principle) by a repeatable objective process (and are thus ideally devoid of opinion). It does not support all needs; gamblers typically require estimates of the odds without experiments.

These two definitions represent two irreconcilable approaches to defining the concept of probability (at least so far). So, there are more fundamental differences between these two areas than whether you can obtain similar estimators or same conclusions in some parametric or nonparametric models.

$\endgroup$
  • 1
    $\begingroup$ I don't think they are completely irreconcilable, a long run frequency is a perfectly reasonable basis for a state of knowledge or belief. $\endgroup$ – Dikran Marsupial Jul 27 '16 at 10:11
  • $\begingroup$ @DikranMarsupial This is actually an open problem in philosophy of science. I see your point, but the reconciliation is not that straightforward. $\endgroup$ – Pollo Anabolico Jul 27 '16 at 10:13
  • $\begingroup$ Thanks Pollo. I agree with you that Frequentists and Bayesians interpret probability differently. As I stated in my preamble, I am focusing not on what they think about probability, but the mechanics of doing inference. If every frequentist inference method has a Bayesian analog (isomorphism, mathematically speaking), then frequentist inference is $functionally$ equivalent to Bayesian inference, regardless of philosophical differences. $\endgroup$ – user75138 Jul 28 '16 at 15:08
  • $\begingroup$ surely frequentist would be functionally equivalent if there were also a frequentist analog for every Bayesian procedure, which can't be true as Bayesians can talk of probabilities for things without long run frequencies, but frequentists can't. $\endgroup$ – Dikran Marsupial Jul 28 '16 at 15:22
  • $\begingroup$ @DikranMarsupial yes, that's why I asked if fequentist are a subset of Bayesian (I acknowledged that the converse is known to be false). $\endgroup$ – user75138 Jul 28 '16 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy