12
$\begingroup$

Do Bayesians ever argue that their approach generalizes the frequentist approach, because one can use non-informative priors and therefore, can recover a typical frequentist model structure?

Can anyone refer me to a place where I can read about this argument, if it is indeed used?

EDIT: This question is maybe phrased not exactly the way I meant to phrase it. The question is: "is there any reference to discussion of the cases in which the Bayesian approach and the frequentist approach overlap/intersect/have something in common through the use of a certain prior?" One example would be using the improper prior $p(\theta) = 1$, but I am pretty sure this is just the tip of the tip of the iceberg.

Improve this question
$\endgroup$
9
  • 2
    $\begingroup$ I recall this argument being made in Greenberg's Introduction to Bayesian Econometrics, but I'm not positive and not sure if there is a better reference. Further, I believe it's not just the choice the prior, but also the confidence in the prior. $\endgroup$
    – John
    Jul 31, 2012 at 20:40
  • 3
    $\begingroup$ There is a good argument that frequentists generalize the Bayes approach! This follows because frequentists are happy to use priors when those are justified (by theory or data) but in addition use methods that Bayesians wouldn't touch. :-) $\endgroup$
    – whuber ♦
    Jul 31, 2012 at 21:02
  • 3
    $\begingroup$ They are completely different approaches starting from the way that Probability is interpreted (see e.g. link). In addition, there is no unique (even less accepted) definition of noninformative prior simply because there is no unique (or accepted) definition of information. Even if the estimators are quantitatively the same, the interpretation of a frequentist estimator and a Bayesian estimator are different. As I mentioned in a previous comment "It is like saying that oranges generalise apples." $\endgroup$
    – user10525
    Aug 1, 2012 at 9:32
  • 1
    $\begingroup$ @Procrastinator I completely agree they don't always intersect. I am looking for arguments in cases where they do. Let me re-frame the question: "is there any reference to a discussion where Bayesian statistics and frequentist statistics overlap in one way or the other through the use of a prior?" One example would be using the improper prior $p(\theta) = 1$. But this is really the tip of the iceberg, I believe. $\endgroup$
    – singelton
    Aug 1, 2012 at 12:34
  • 1
    $\begingroup$ @Procrastinator yes, thanks! that's exactly the kind of discussion I am looking for (though, I am guessing it is still the tip of the iceberg). I just need to find a book that does it thoroughly, and I was unable to find one. I will keep looking. thanks again. (most books focus either on the frequentist approach or the Bayesian approach, but don't compare the two the way you did.) $\endgroup$
    – singelton
    Aug 1, 2012 at 12:46

5 Answers 5

Reset to default
11
$\begingroup$

I have seen two arguments advanced that Bayesian analysis is a generalization of a frequentist analysis. Both were somewhat tongue-in-cheek, and more getting people to recognize the assumptions about regression models by using priors as a context.

Argument 1: Frequentist analysis is Bayesian analysis with a purely uninformative prior centered on zero (yes, it doesn't matter where its centered, but ignore that). This provides both the context for which a Bayesian might extract the results of a frequentist analysis, explains why you can get away with using some "Bayesian" techniques like MCMC to extract frequentist estimates in situations where say, maximum likelihood convergence is tough, and gets people to recognize that when they say "The data speak for themselves" and the like, what they're actually saying is that beforehand, all values are equally likely.

Argument 2: Any regression term you don't include in a model has, in effect, been assigned a prior centered on zero with no variance. This one isn't so much a "Bayesian analysis is a generalization" as much as "There are priors everywhere, even in your frequentist models" argument.

Improve this answer
$\endgroup$
6
  • 3
    $\begingroup$ +1 Argument 2 is interesting. Two comments on Argument 1: 1. I'd say flat priors instead of uninformative (the latter is a misnomer, if ever there was one). 2. There is no need to talk about priors in order to motivate the use of MCMC in frequentist analysis - there is nothing inherently Bayesian about this numerical technique! $\endgroup$
    – MånsT
    Aug 1, 2012 at 10:20
  • $\begingroup$ thank you EpiGrad. Do you have any references that discuss the two arguments you mentioned? $\endgroup$
    – singelton
    Aug 1, 2012 at 11:25
  • 1
    $\begingroup$ +1 Okay as long as people realize that it is tongue in cheek to get a point across. But please don't take it seriously! $\endgroup$
    – Michael R. Chernick
    Aug 1, 2012 at 13:22
  • $\begingroup$ @MånsT - Agreed about MCMC not needing a justification for use, but I find it exists in peoples mind as something in the Bayesian realm, rather than a purely numerical technique. This helps push them off that. $\endgroup$
    – Fomite
    Aug 1, 2012 at 17:06
  • $\begingroup$ @bayesianOrFrequentist Not really no. $\endgroup$
    – Fomite
    Aug 1, 2012 at 17:06
6
$\begingroup$

The short answer is probably "yes--and you don't even need a flat prior for this argument to hold."

For example, the Maximum A Posteriori (MAP) estimate is a generalization of maximum likelihood that includes a prior, and there are frequentist approaches that are analytically equivalent to finding this value. The frequentist relabels "the prior" as a "constraint" or "penalty" on the likelihood function, and gets the same answer. So frequentists and Bayesians can both point to the same thing as being their best parameter estimate, even if the philosophies are different. Section 5 of this frequentist paper is one example where they're equivalent.

The longer answer is more like "yes, but there are often other aspects of the analysis that distinguish the two approaches. Still, even these distinctions aren't necessarily iron-clad in many cases."

For example, while Bayesians do sometimes use the MAP estimate (posterior mode) when it's convenient, they typically emphasize the posterior mean instead. On the other hand, the posterior mean also has a frequentist analogue, called the "bagged" estimate (from "bootstrap aggregating") that can be almost indistinguishable (see this pdf for an example of this argument). So that's not really a "hard" distinction either.

In practice, all this means that even when a frequentist does something that a Bayesian would consider totally illegal (or vice versa), there is often (at least in principle) an approach from the other camp that would give nearly the same anser.

The main exception is that some models are really hard to fit from a frequentist perspective, but that's more of a practical issue than a philosophical one.

Improve this answer
$\endgroup$
5
  • $\begingroup$ thanks David. Your answer is useful. I am also looking for a reference that discusses this point at length. I want to see what the argument of Bayesians is about non-informative priors and the way they can be reduced to the frequentist approach. I perfectly understand the technical point behind it (for example, if you just multiply your likelihood by 1... you are going to get your likelihood :-)), but I am looking for a more decent discussion. $\endgroup$
    – singelton
    Jul 31, 2012 at 20:53
  • 1
    $\begingroup$ I am finding that many young people do not know the history or understand the essence of the Bayesian paradigm. To call it a generalization of the frequentist approach really misrepresents the comparison of these paradigms. Taking Procrastinators comment and putting it in a slightly different way, I would say this is like saying that an apple is just an oversized orange, $\endgroup$
    – Michael R. Chernick
    Aug 1, 2012 at 13:19
  • $\begingroup$ @DavidJHarris I didn't like your answer. Technically the relationships you point to are legitimate but to say "yes" in the short answer gives the wrong impression. I don't think Bayesians would want to call their paradigm a generalization of frequntist statistics. The terms fully Bayesian, empirical Bayesian and possibly distinguish Bayesian-related paradigms but I think Bayesians might object to calling these branches of the Bayesian paradigm. $\endgroup$
    – Michael R. Chernick
    Aug 1, 2012 at 13:30
  • 2
    $\begingroup$ @MichaelChernick Point taken. I didn't mean to imply that all of Bayesian statistics and philosophy has close frequentist analogues and vice-versa, only that one can often find a method that will accomplish the same job from either camp, and that the Bayesian approach tends to be the more flexible of the two. Perhaps I should have emphasized that, even when the parameter estimates you get from the two schools are identical, they should still be interpreted differently, as Procrastinator pointed out elsewhere. $\endgroup$
    – David J. Harris
    Aug 1, 2012 at 16:32
  • $\begingroup$ @DavidJHarris. I agree with everything you say but only take exception to the use of the term generalization. $\endgroup$
    – Michael R. Chernick
    Aug 2, 2012 at 2:16
3
$\begingroup$

Edwin Jaynes was one of the best at highlighting the connections between bayesian and frequentist inference. His paper confidence intervals vs bayesian intervals (google search brings it up) as a very thorough comparison - and I think a fair one.

Small area estimation is another area where ML/REML/EB/HB answers tend to be close.

Improve this answer
$\endgroup$
2
$\begingroup$

Many of these comments assume that "frequentist" means "maximum likelihood estimation." Some people have a different definition: "frequentist" means a type of analysis of the long-term inferential properties of any inference method -- whether it's Bayesian, or method-of-moments, or maximum likelihood, or something couched in non-probabilistic terms (e.g SVM's), etc.

Improve this answer
$\endgroup$
1
$\begingroup$

I would like to hear from Stephane or some other Bayesian expert on this. I would say no because it is a different approach not a generalization. In another context this has been argued here before. Don't think that just because flat priors produce results close to maximum likelihood that a Bayesian method with a flat prior is frequentist! I think that would be a false presumption that would lead you to think that by making the prior arbitrary you are generalizing to other possible priors. I don't think that way and I am pretty sure most Bayesian don't either.

So some people do argue it but I don't think they should be classified as Bayesians

although Stephane has pointed out the difficulty with strong classification. So strictly speaking if the word is ever then I guess it might depend on how you define Bayesian.

Improve this answer
$\endgroup$
3
  • $\begingroup$ (+1) They are completely different approaches. It is like saying that oranges generalise apples. $\endgroup$
    – user10525
    Aug 1, 2012 at 9:27
  • 5
    $\begingroup$ Eating lots of oranges and no apples makes one thinks so. $\endgroup$
    – M. Toya
    Aug 1, 2012 at 10:35
  • $\begingroup$ this is true, although maximum likelihood is one of the few general procedures for doing frequentist inference. So it invariably will be over-represented in general discussions about frequentist methods. I'm surprised survey sampling hasn't been mentioned such as GREG. $\endgroup$
    – probabilityislogic
    Aug 1, 2012 at 23:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.