5
$\begingroup$

I realize the methodology pursued by the Frequentist and Bayesian camps generally differ. However, one method of estimation that they do share is optimization of a certain function:

  • Frequentists maximize the likelihood function, giving the Max. Likelihood (ML) estimator.
  • Bayesians maximize the posterior function, giving the Max A-Posteriori (MAP) estimator.

Both functions will typically have been constructed using Baye's rule/theorem, which is universally agreed upon, and which might have been applied once (in "batch mode") or multiple times iteratively.

Similarly, both Frequentists and Bayesians will deduce their interval (confidence/credibility) from this function.

So if the prior is uninformative (assuming we can formulate such a prior), there should be no distinction between the "results" obtained by Bayesians and Frequentists, even though the interpretation of said results will be different.

If this is right, then the only practical difference between Bayesians and Frequentists is the prior. Is this true?

Edit:

Actually, the optimization bit of my question is a bit misleading, as it is only a specific example of differences between Bayesian and Frequentist thinking. My question could be posed simply as the difference simply between the likelihood function and the posterior. For example, would frequentists ever use MCMC to calculate the likelihood function?

Edit, 10 years on: This was a confused question from a confused student. To be fair, the topic tends to confuse the uninitiated. Thanks to everyone who participated in the discussions.

Improve this question
$\endgroup$
13
  • 1
    $\begingroup$ It depends on the definition of noninformative prior you use. If you use a flat prior on a domain that contains the MLE, then this coincides with the MAP but if you use a different sort of noninformative prior, then they may differ. $\endgroup$
    – user10525
    Commented Jun 1, 2012 at 17:16
  • 1
    $\begingroup$ Regarding interval estimation, Bayesian and frequentist intervals have similar properties if you use matching priors, but not necessarily under the use of other sorts of noninformative priors. For more information about the differences between these two approaches, take a look at this questions 1, 2 $\endgroup$
    – user10525
    Commented Jun 1, 2012 at 17:30
  • 1
    $\begingroup$ Indeed, Data cloning is an example of an MCMC method used for maximising the likelihood function. Also, both approaches get benefited by some nonparametric methods, see 1, 2. $\endgroup$
    – user10525
    Commented Jun 1, 2012 at 18:11
  • $\begingroup$ ML is not the only frequentist estimation procedure. Many estimation procedures can be justified because they are Bayes according to some (perhaps arbitrary) prior; a theorem says that (under certain regularity assumptions) such procedures are admissible. (One might say that every frequentist is a Bayesian when a defensible prior can be found, but also a frequentist would hesitate--on the same grounds--to use an "uninformative" prior.) But other frequentist procedures--notably certain minimax estimators--might not have any Bayes counterparts at all. $\endgroup$
    – whuber ♦
    Commented Jun 1, 2012 at 19:04
  • 1
    $\begingroup$ @StéphaneLaurent, ok so where exactly do they deduce it from? $\endgroup$
    – Patrick
    Commented Jun 2, 2012 at 14:31

2 Answers 2

Reset to default
5
$\begingroup$

The maximum aposteriori (MAP) approach isn't really fully Bayesian, ideally inference should involve marginalising over the whole posterior. Optimisation is the root of all evil in statistics; it is difficult to over-fit if you don't optimise! ;o) So the practical difference between Bayesian and frequentist runs rather more deeply if you opt for a fully Bayesian solution, although there will often be a prior for which the result is numerically the same as the frequentist approach.

However, the credible interval and confidence interval are answers to different questions, and shouldn't be considered interchangable, even if they happen to be numerically the same. Treating a frequentist confidence interval as if it was a Bayesian credible interval can lead to problems of interpretation.

UPDATE "My question could be posed simply as the difference simply between the likelihood function and the posterior."

No, the definitions of probability are different, this means that even if the solutions are numerically identical, this doesn't imply they mean the same thing. This is a practical issue as well as a conceptual one, as the correct intepretation of the result depends on the definition of probability.

Tongue in cheek, if the answer to the question was "yes", it would imply that frequentists were merely Bayesians that always used flat (often improper) priors for everything. I doubt many frequentists would agree with that! ;o)

Improve this answer
$\endgroup$
3
  • $\begingroup$ Yes, if you've done all the hard work involved in calculating the posterior, you don't want to thrown away information by reducing it to the MAP. I guess my question can be reformulated simply in terms of posterior func. vs likelihood func., instead of MAP vs ML. I am aware that the credibility and confidence intervals are answers to different questions. However, they can both be used as interval estimates for the parameter, which makes them comparable. $\endgroup$
    – Patrick
    Commented Jun 1, 2012 at 17:47
  • $\begingroup$ However different things are meant by "interval estimate" in each framework. All too often a frequentist interval is interpreted as being an interval that contains the true value with high probability, which is not supported by the frequentist analysis, and as Jaynes demonstrated isn't necessarily even true. Rather than adopting one framework or the other, the best thing to do is to be comfortable with both, and use the framework that most directly answers the question you really want answered. $\endgroup$
    – Dikran Marsupial
    Commented Jun 1, 2012 at 18:02
  • 1
    $\begingroup$ @DikranMarsupial +1 I really like your answer and also your followup comment. I think the war between the Bayesian's and frequentists is over and nobody won. A lot of statisticians including Brad Efron are adopting the position you describe in last sentence of the statement above. $\endgroup$
    – Michael R. Chernick
    Commented Jun 1, 2012 at 19:18
5
$\begingroup$

I would agree that roughly speaking you are right. Priors noninformative or not will lead to different solutions. The solutions will converge when the data dominates the prior. Also Jeffreys needed improper priors in some cases to match Bayesian results with frequentist results. The real difference and the controversy is philosophical. Frequentists want objectivity. The prior brings in subjective opinion. Bayesians following the teachings of Di Finetti believe that probability is subjective. For a true Bayesian priors should be informative. The other point also related to the differing concepts of probability is that probability can be assigned to an unknown parameter according to Bayesians while frequentist think strictly in term of probability spaces as given in the theory developed by Kolmogorov and von Mises. For frequentists it is only the random avriables that you can define on the probaility space that have probabilities associated with their outcomes. So the probability of getting a head on a coin toss is 1/2 because repeated flipping leads to a relative frequence of heads that converges to 1/2 as the sample size approaches infinity.

For frequentists Bayes' theorem applies to events which are measureable sets in a probabilty space. The Bayesians apply it to parameters as if the parameter was a random variable. That is the frequentists objection to Bayesian methods. The Bayesians object to the frequenttist approach because it lacks a property called coherence. I will not go into that here but you can look up the definiton on the internet or read Dennis Lindley's books.

Improve this answer
$\endgroup$
18
  • 3
    $\begingroup$ +1, however I would disagree that Bayesianism is necessarily subjective (I agree with Jayenes on that one) or that frequentism is completely objective (there are often assumptions that are equivalent to having a prior belief about the model, but they are not a formal part of the framework, but exist nevetheless). Also I don't see why noninformative priors are non-Bayesian, they encode the prior knowledge that you know you don't know something. As usual, this doesn't imply that Bayesianism is better than frequentism or vice versa, just as a flat screwdriver is not better than a pozidrive one. $\endgroup$
    – Dikran Marsupial
    Commented Jun 1, 2012 at 17:31
  • $\begingroup$ @DikranMarsupial I think there are various camps in both the Bayesian and frequentists schools. But I think what I would call the pure Bayesians are those that follow Di Finetti's subjective view of probability. The purpose of the prior is to include subjective information prior to collecting data. Pure frequentists follow von Mises approach to probability. Where should we put the empirical Bayesians? Do the belong to the Bayesian school or in their attempt to inform the prior with the data are the more in the frequentists school. $\endgroup$
    – Michael R. Chernick
    Commented Jun 1, 2012 at 17:46
  • 1
    $\begingroup$ Fisher was the first "fiducialist." A school whose following is almost nonexistent now. But his school wanted to do inference based on a form of inverse probability that involved the likelihood function but not the prior yet differed from the approach to inference due to Neyman. My point was not really to strictly define the Bayesian and frequentist camps but rather to show the OP (Patrick) that it isn't quite as simple as he makes it out to be. $\endgroup$
    – Michael R. Chernick
    Commented Jun 1, 2012 at 17:51
  • 1
    $\begingroup$ I agree about the various camps, if you feel that only followers of di Finetti are true Bayesians then perhaps it would be better state that it is your opinion, rather than state it as a fact, as followers of Jayenes for instance would not be in agreement, and are no less pure Bayesians than those of di Finetti, neither have a claim to be the one true school, nor to be the final word. In my experience uninformative priors are extremely useful as a guard against jumping to conclusions by ingoring your ignorance of some aspect of the system. $\endgroup$
    – Dikran Marsupial
    Commented Jun 1, 2012 at 17:54
  • $\begingroup$ I would include anyone who viewed probability as being a measure of the state of knowledge as being a Bayesian, and those who strictly limit it to long run frequencies as being frequentists. That seems the most crucial distinction between the two approaches. I agree with most of what you wrote (hence the +1) I was just adding some caveats where it was over-specific. I had an interesting discussion about fiducialism with a frequentist colleague a while back, it seems to me that it was a tacit admission that the quest for objectivity placed uncomfortable limits on what could be addressed. $\endgroup$
    – Dikran Marsupial
    Commented Jun 1, 2012 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.