When (and why) do Bayesians reject valid Bayesian methods? [closed]

Question

From what I have read and from answers to other questions I have asked here, many so-called frequentist methods correspond mathematically (I don't care if they correspond philosophically, I only care whether it corresponds mathematically) to special cases of so-called Bayesian methods (for those who object to this, see the note at the bottom of this question). This answer to a related question (not mine) supports this conclusion:

Most Frequentist methods have a Bayesian equivalent that in most circumstances will give essentially the same result.

Note that in what follows, being mathematically the same means giving the same result. If you characterize two methods which can be proved to always give the same results as being "different", that is your right, but that is a philosophical judgment, not a mathematical one nor a practical one.

Many people who self-describe as "Bayesians", however, seem to reject using maximum likelihood estimation under any circumstances, even though it is a special case of (mathematically) Bayesian methods, because it is a "frequentist method". Apparently Bayesians also use a restricted/limited number of distributions compared to frequentists, even though those distributions would also be mathematically correct from a Bayesian viewpoint.

Question: When and why do Bayesians reject methods which are mathematically correct from a Bayesian viewpoint? Is there any justification for this which isn't "philosophical"?

Background/Context: The following are quotes from answers and comments to a previous question of mine on CrossValidated:

The mathematical basis for the Bayesian vs frequentist debate is very simple. In Bayesian statistics the unknown parameter is treated as a random variable; in frequentist statistics it is treated as a fixed element...

From the above I would have concluded that (mathematically speaking) Bayesian methods are more general than frequentist ones, in the sense that frequentist models satisfy all of the same mathematical assumptions as Bayesian ones, but not vice versa. However, the same answer argued that my conclusion from the above was incorrect (emphasis in what follows is mine):

Although the constant is a special case of a random variable, I would hesitate to conclude that Bayesianism is more general. You would not get frequentist results from Bayesian ones by simply collapsing the random variable to a constant. The difference is more profound...

Going to personal preferences... I do not like that Bayesian statistics uses quite a restricted subset of available distributions.

Another user, in their answer, stated the opposite, that Bayesian methods are more general, although oddly enough the best reason I could find for why this might be the case was in the previous answer, given by someone trained as a frequentist.

The mathematical consequence is that Frequentists think the basic equations of probability only sometimes apply, and Bayesians think they always apply. So they view the same equations as correct, but differ on how general they are... Bayesian is strictly more general than Frequentist. Since there can be uncertainty about any fact, any fact can be assigned a probability. In particular, if the facts you're working on is related to real world frequencies (either as something you're predicting or part of the data) then Bayesian methods can consider and use them just as they would any other real world fact. Consequently any problem Frequentists feel their methods apply to Bayesians can also work on naturally.

From the above answers, I have the impression that there are at least two different definitions of the term Bayesian commonly in use. The first I would call "mathematically Bayesian" which encompasses all methods of statistics, since it includes parameters which are constant RVs and those which are not constant RVs. Then there is "culturally Bayesian" which rejects some "mathematically Bayesian" methods because those methods are "frequentist" (i.e. out of personal animosity to the parameter sometimes being modeled as a constant or frequency). Another answer to the aforementioned question also seems to support this conjecture:

It is also of note that there are plenty divides between the models used by the two camps that is more related to what has been done than what can be done (i.e. many models that are traditionally used by one camp can be justified by the other camp).

So I guess another way to phrase my question would be the following: Why do cultural Bayesians call themselves Bayesians if they reject many mathematically Bayesian methods? And why do they reject these mathematically Bayesian methods? Is it personal animosity for the people who most often use those particular methods?

Edit: Two objects are equivalent in a mathematical sense if they have the same properties, irrespective of how they are constructed. For example, I can think of at least five different ways to construct the imaginary unit $i$. Nevertheless, there are not at least five different "schools of thought" about the study of imaginary numbers; in fact, I believe there is only one, which is that group which studies their properties. To those who object that getting a point estimate using maximum likelihood is not the same thing as getting a point estimate using maximum a priori and a uniform prior because the calculations involved are different, I concede that they are different in a philosophical sense, but to the extent that they always give the same values for the estimate, they are mathematically equivalent, because they have the same properties. Maybe the philosophical difference is relevant to you personally, but it isn't relevant to this question.

Note: This question originally had an incorrect characterization of MLE estimation and MAP estimation with a uniform prior.

Answer 1

I would like to correct an erroneous assumption in the original post, a mistake which is relatively common. The OP says:

From what I have read and from answers to other questions I have asked here, maximum likelihood estimation corresponds mathematically (I don't care if it corresponds philosophically, I only care whether it corresponds mathematically) to maximum a priori estimation using a uniform prior (for those who object to this, see the note at the bottom of this question).

And the note at the bottom of the post says:

Two objects are equivalent in a mathematical sense if they have the same properties, irrespective of how they are constructed. [...]

My objection is that, philosophy aside, maximum likelihood estimation (MLE) and maximum-a-posteriori (MAP) estimation do not have the same mathematical properties.

Crucially, MLE and MAP transform differently under (nonlinear) reparametrization of the space. This happens because MLE has a "flat prior" in every parametrization, whereas MAP doesn't (the prior transforms as a probability density, so there is a Jacobian term).

The definition of a mathematical object includes how the object behaves under operators such as transformation of variables (e.g., see the definition a tensor).

In conclusion, MLE and MAP are not the same thing, neither philosophically nor mathematically; this is not an opinion.

Answer 2

Personally I am a "pragmatist" rather than a "frequentist" or a "Bayesian", so I cannot claim to speak for any camp.

That said, I think the distinction you are alluding to is probably not so much MLE vs. MAP, but between point estimates vs. estimating posterior PDFs. As a scientist working in a field with sparse data and large uncertainties, I can sympathize with not wanting to put too much confidence on "best guess" results which may be misleading, resulting in overconfidence.

A related practical distinction is between parametric vs. non-parametric methods. So for example I think that both Kalman filtering and Particle filtering would be accepted as Recursive Bayesian Estimation. But the Gaussian assumption of Kalman filtering (a parametric method) can give very misleading results if the posterior is not unimodal. To me these kinds of engineering examples highlight where differences are neither philosophical nor mathematical, but manifest in terms of practical results (i.e. will your autonomous vehicle crash?). For the Bayesian enthusiasts I am familiar with, this "see what works" engineering-style attitude seems to be predominant ... not sure if this is true more broadly.

Answer 3

Many people who self-describe as "Bayesians", however, seem to reject using maximum likelihood estimation under any circumstances, even though it is a special case of (mathematically) Bayesian methods, because it is a "frequentist method".

Such people would be rejecting MLE as a general method for making point estimates. In particular cases where they had reason to use a uniform prior & wanted to make a maximum a posteriori estimate they wouldn't be at all bothered by the coincidence of their calculations with MLE.

Apparently Bayesians also use a restricted/limited number of distributions compared to frequentists, even though those distributions would also be mathematically correct from a Bayesian viewpoint.

Perhaps sometimes, to make their calculations easier, but not from any point of principle.

I have the impression that there are at least two different definitions of the term Bayesian commonly in use. The first I would call "mathematically Bayesian" which encompasses all methods of statistics, since it includes parameters which are constant RVs and those which are not constant RVs. Then there is "culturally Bayesian" which rejects some "mathematically Bayesian" methods because those methods are "frequentist" (i.e. out of personal animosity to the parameter sometimes being modeled as a constant or frequency).

There are certainly distinctions to be made between different approaches to Bayesian inference, but not this one. If there's a sense in which Bayesianism is more general, it's in the willingness to apply the concept of probability to epistemic uncertainty about parameter values & not just the aleatory uncertainty of the data-generating process which is all that frequentism concerns itself with. Frequentist inference is not a special case of Bayesian inference & none of the answers or comments at Is there any mathematical basis for the Bayesian vs frequentist debate? are implying that it is. If in a Bayesian approach you were to consider the parameter a constant random variable, you'd obtain the same posterior whatever the data are—& to say it's constant but you don't know what value it takes wouldn't be to say anything worth saying. The frequentist approach takes an entirely different tack & doesn't involve the calculation of posterior distributions at all.