From what I have read and from answers to other questions I have asked here, many so-called frequentist methods correspond mathematically (I don't care if they correspond philosophically, I only care whether it corresponds mathematically) to special cases of so-called Bayesian methods (for those who object to this, see the note at the bottom of this question). This answer to a related question (not mine) supports this conclusion:

Most Frequentist methods have a Bayesian equivalent that in most circumstances will give essentially the same result.

Note that in what follows, being mathematically the same means giving the same result. If you characterize two methods which can be proved to always give the same results as being "different", that is your right, but that is a philosophical judgment, not a mathematical one nor a practical one.

Many people who self-describe as "Bayesians", however, seem to reject using maximum likelihood estimation under any circumstances, even though it is a special case of (mathematically) Bayesian methods, because it is a "frequentist method". Apparently Bayesians also use a restricted/limited number of distributions compared to frequentists, even though those distributions would also be mathematically correct from a Bayesian viewpoint.

Question: When and why do Bayesians reject methods which are mathematically correct from a Bayesian viewpoint? Is there any justification for this which isn't "philosophical"?

enter image description here

Background/Context: The following are quotes from answers and comments to a previous question of mine on CrossValidated:

The mathematical basis for the Bayesian vs frequentist debate is very simple. In Bayesian statistics the unknown parameter is treated as a random variable; in frequentist statistics it is treated as a fixed element...

From the above I would have concluded that (mathematically speaking) Bayesian methods are more general than frequentist ones, in the sense that frequentist models satisfy all of the same mathematical assumptions as Bayesian ones, but not vice versa. However, the same answer argued that my conclusion from the above was incorrect (emphasis in what follows is mine):

Although the constant is a special case of a random variable, I would hesitate to conclude that Bayesianism is more general. You would not get frequentist results from Bayesian ones by simply collapsing the random variable to a constant. The difference is more profound...

Going to personal preferences... I do not like that Bayesian statistics uses quite a restricted subset of available distributions.

Another user, in their answer, stated the opposite, that Bayesian methods are more general, although oddly enough the best reason I could find for why this might be the case was in the previous answer, given by someone trained as a frequentist.

The mathematical consequence is that Frequentists think the basic equations of probability only sometimes apply, and Bayesians think they always apply. So they view the same equations as correct, but differ on how general they are... Bayesian is strictly more general than Frequentist. Since there can be uncertainty about any fact, any fact can be assigned a probability. In particular, if the facts you're working on is related to real world frequencies (either as something you're predicting or part of the data) then Bayesian methods can consider and use them just as they would any other real world fact. Consequently any problem Frequentists feel their methods apply to Bayesians can also work on naturally.

From the above answers, I have the impression that there are at least two different definitions of the term Bayesian commonly in use. The first I would call "mathematically Bayesian" which encompasses all methods of statistics, since it includes parameters which are constant RVs and those which are not constant RVs. Then there is "culturally Bayesian" which rejects some "mathematically Bayesian" methods because those methods are "frequentist" (i.e. out of personal animosity to the parameter sometimes being modeled as a constant or frequency). Another answer to the aforementioned question also seems to support this conjecture:

It is also of note that there are plenty divides between the models used by the two camps that is more related to what has been done than what can be done (i.e. many models that are traditionally used by one camp can be justified by the other camp).

So I guess another way to phrase my question would be the following: Why do cultural Bayesians call themselves Bayesians if they reject many mathematically Bayesian methods? And why do they reject these mathematically Bayesian methods? Is it personal animosity for the people who most often use those particular methods?

Edit: Two objects are equivalent in a mathematical sense if they have the same properties, irrespective of how they are constructed. For example, I can think of at least five different ways to construct the imaginary unit $i$. Nevertheless, there are not at least five different "schools of thought" about the study of imaginary numbers; in fact, I believe there is only one, which is that group which studies their properties. To those who object that getting a point estimate using maximum likelihood is not the same thing as getting a point estimate using maximum a priori and a uniform prior because the calculations involved are different, I concede that they are different in a philosophical sense, but to the extent that they always give the same values for the estimate, they are mathematically equivalent, because they have the same properties. Maybe the philosophical difference is relevant to you personally, but it isn't relevant to this question.

Note: This question originally had an incorrect characterization of MLE estimation and MAP estimation with a uniform prior.

closed as unclear what you're asking by Juho Kokkala, Greenparker, Nick Cox, COOLSerdash, Peter Flom Aug 22 '16 at 11:20

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 8
    (-1) This question is based on false assumptions. MLE does not correspond to 'using uniform prior' but to using uniform prior and selecting the mode of the posterior distribution (so, MAP with uniform prior). When using MLE, the parameter is not considered a random variable, so constructions like $Pr(\theta \in [0,1] \mid y)$, or integrals over $\theta \mid y$ are mathematically not meaningful. – Juho Kokkala Aug 21 '16 at 5:35
  • 3
    I do not recall any Bayesians who either reject anything that is non-Bayesian by name, or who use limited number of distributions. One could easily replace "Bayesians" with "frequentists" in your question and ask about why frequentists reject everything that is non-frequentist, and why do they use limitted number of distributions (basically, normal distribution everywhere) -- the resulting question would be the same ill-defined as yours. I also agree with @JuhoKokkala that MLE $\neq$ using uniform prior even though their point estimates may correspond. – Tim Aug 21 '16 at 19:36
  • 5
    MLE and MAP do not have the same mathematical properties. If you reparametrize your variables, MLE and MAP transform differently (because MLE has a "flat prior" in every parametrization, MAP doesn't). The definition of a mathematical object includes how the object behaves under operators such as transformation of variables (e.g., see the definition a tensor). So they are not the same thing. – lacerbi Aug 21 '16 at 20:38
  • 2
    I will make it a (short) answer, since it's surprising that nobody has mentioned this so far. I also had to explain it many times in the past, since it's a subtlety that can be easily missed. – lacerbi Aug 21 '16 at 22:10
  • 3
    Have you ever played draughts with a chess set? It can happen from time to time that you find yourself in a valid chess position & can make a legal chess move that's also a legal draughts move. Of course what would be a good chess move won't always be a good draughts move. And you won't eschew making a good draughts move just because it's also a chess move. This is rather different from describing a chess game in French rather than English, or from rotating the board so that black squares become white or swapping the initial positions & rules governing the ... – Scortchi Aug 22 '16 at 9:11
up vote 10 down vote accepted

I would like to correct an erroneous assumption in the original post, a mistake which is relatively common. The OP says:

From what I have read and from answers to other questions I have asked here, maximum likelihood estimation corresponds mathematically (I don't care if it corresponds philosophically, I only care whether it corresponds mathematically) to maximum a priori estimation using a uniform prior (for those who object to this, see the note at the bottom of this question).

And the note at the bottom of the post says:

Two objects are equivalent in a mathematical sense if they have the same properties, irrespective of how they are constructed. [...]

My objection is that, philosophy aside, maximum likelihood estimation (MLE) and maximum-a-posteriori (MAP) estimation do not have the same mathematical properties.

Crucially, MLE and MAP transform differently under (nonlinear) reparametrization of the space. This happens because MLE has a "flat prior" in every parametrization, whereas MAP doesn't (the prior transforms as a probability density, so there is a Jacobian term).

The definition of a mathematical object includes how the object behaves under operators such as transformation of variables (e.g., see the definition a tensor).

In conclusion, MLE and MAP are not the same thing, neither philosophically nor mathematically; this is not an opinion.

  • I think perhaps I've missed your point. Is it possible to parameterize a model such that the point estimates from MLE aren't equal to those from MAP with a uniform prior? (Clearly, in the MAP case, the prior has to be uniform with respect to the current parametrization in order for the equality to work. If you reparametrize the model without changing the prior, then it will in general no longer be uniform.) – Kodiologist Aug 22 '16 at 16:30
  • 1
    @Kodiologist: the OP was stating that MAP and MLE are identical "mathematical objects". They are not. Distinct mathematical objects can be equal in a subspace (e.g., in a given parametrization) but that does not make them identical. You could say "I don't care about other parametrizations" but, well, then you are imposing a strong practical restriction, it's not "merely" a philosophical point anymore as the OP was originally arguing for. – lacerbi Aug 22 '16 at 16:57

Personally I am a "pragmatist" rather than a "frequentist" or a "Bayesian", so I cannot claim to speak for any camp.

That said, I think the distinction you are alluding to is probably not so much MLE vs. MAP, but between point estimates vs. estimating posterior PDFs. As a scientist working in a field with sparse data and large uncertainties, I can sympathize with not wanting to put too much confidence on "best guess" results which may be misleading, resulting in overconfidence.

A related practical distinction is between parametric vs. non-parametric methods. So for example I think that both Kalman filtering and Particle filtering would be accepted as Recursive Bayesian Estimation. But the Gaussian assumption of Kalman filtering (a parametric method) can give very misleading results if the posterior is not unimodal. To me these kinds of engineering examples highlight where differences are neither philosophical nor mathematical, but manifest in terms of practical results (i.e. will your autonomous vehicle crash?). For the Bayesian enthusiasts I am familiar with, this "see what works" engineering-style attitude seems to be predominant ... not sure if this is true more broadly.

  • 1
    Whether the noise is modeled Gaussian or from another distribution is not what determines whether a method is parametric or non-parametric. – Cliff AB Aug 21 '16 at 3:57
  • 1
    I was thinking of particle filtering vs. Kalman filtering. – GeoMatt22 Aug 21 '16 at 4:10
  • 1
    @CliffAB I edited my answer to hopefully fix the unintended implication that "Gaussian <==> parametric" – GeoMatt22 Aug 21 '16 at 4:16
  • 2
    In my experience (not at all comprehensive!), the books aimed at engineers in "tech" areas tend to be more like this. Things like robotics and other real-time/robust applications tend to find out quickly when things are not working. It is probably nominally more Bayesian, but Sebastian Thrun's Probabilistic Robotics was enlightening to me. He is the Udacity guy. – GeoMatt22 Aug 21 '16 at 20:24
  • 2
    I have not studied this area at all, but my impression is that much of classical Reliability Engineering uses "frequentist" approaches, so this may also be an area with pragmatic texts? – GeoMatt22 Aug 21 '16 at 20:44

Many people who self-describe as "Bayesians", however, seem to reject using maximum likelihood estimation under any circumstances, even though it is a special case of (mathematically) Bayesian methods, because it is a "frequentist method".

Such people would be rejecting MLE as a general method for making point estimates. In particular cases where they had reason to use a uniform prior & wanted to make a maximum a posteriori estimate they wouldn't be at all bothered by the coincidence of their calculations with MLE.

Apparently Bayesians also use a restricted/limited number of distributions compared to frequentists, even though those distributions would also be mathematically correct from a Bayesian viewpoint.

Perhaps sometimes, to make their calculations easier, but not from any point of principle.

I have the impression that there are at least two different definitions of the term Bayesian commonly in use. The first I would call "mathematically Bayesian" which encompasses all methods of statistics, since it includes parameters which are constant RVs and those which are not constant RVs. Then there is "culturally Bayesian" which rejects some "mathematically Bayesian" methods because those methods are "frequentist" (i.e. out of personal animosity to the parameter sometimes being modeled as a constant or frequency).

There are certainly distinctions to be made between different approaches to Bayesian inference, but not this one. If there's a sense in which Bayesianism is more general, it's in the willingness to apply the concept of probability to epistemic uncertainty about parameter values & not just the aleatory uncertainty of the data-generating process which is all that frequentism concerns itself with. Frequentist inference is not a special case of Bayesian inference & none of the answers or comments at Is there any mathematical basis for the Bayesian vs frequentist debate? are implying that it is. If in a Bayesian approach you were to consider the parameter a constant random variable, you'd obtain the same posterior whatever the data are—& to say it's constant but you don't know what value it takes wouldn't be to say anything worth saying. The frequentist approach takes an entirely different tack & doesn't involve the calculation of posterior distributions at all.

  • "The frequentist approach takes an entirely different tack & doesn't involve the calculation of posterior distributions at all" -- that isn't my point though. I am not talking about philosophical intent, I am talking about mathematical equivalence. Someone could say that they are a "subtractivist" because they only add and subtract positive numbers but refuse to use negative numbers, which is "negativist". Philosophically that might be the case, but mathematically speaking subtracting a positive number is the same thing as adding a negative one. – Chill2Macht Aug 21 '16 at 18:26
  • What I am trying to say is that "mathematically Bayesian" would be both applying and not applying the concept of probability to epistemic uncertainty about parameter values. "Culturally Bayesian" would be only applying (and never not applying) the concept of probability to epistemic uncertainty about parameter values. "Frequentist" would be only not applying (and never applying) probability to epistemic uncertainty about parameter values. What I am saying is that both "Bayesian inference=culturally Bayesian" and "frequentist" seem like a special case based on what people are saying. – Chill2Macht Aug 21 '16 at 18:31
  • Anyway I guess I will try to read van der Vaart's Asymptotic Statistics before commenting further on frequentist statistics, but having read already Casella and Berger and zero Bayesian textbooks I don't understand the statement that "the frequentist approach takes an entirely different tack" from applying the concept of probability to "just the aleatory uncertainty of the data-generating process", since it seems to contradict the other parts of what you wrote. – Chill2Macht Aug 21 '16 at 18:56
  • 2
    (1) If your point's that frequentist and Bayesian procedures are mathematically equivalent, but merely described in different terms, then it's just not true. They coincide on occasion - just as subtractivist & negativist subtraction coincides until you come to work out $3-5$. (2) If I may read "both applying and not applying" as "sometimes applying, sometimes not applying" then "mathematically Bayesian" is just sometimes Bayesian, sometimes frequentist, & "culturally Bayesian" just Bayesian. (3) I'd recommend Cox (2006), Principles of Statistical Inference, Geisser (2006), ... – Scortchi Aug 21 '16 at 20:30
  • 2
    Modes of Parametric Statistical Inference, & Barnett (1999), Comparative Statistical Inference. (4) The frequentist approach considers only the probability of the data under given parameter values; the Bayesian approach conditions on the observed data to obtain a posterior. – Scortchi Aug 21 '16 at 20:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.