Skip to main content
3 of 8
Expanded answer to include proper definition of Bayes estimator
pglpm
  • 1.3k
  • 9
  • 20

Xi'an's answer explains the core point very clearly and concisely. The present answer wants to spell out how the maximum-likelihood and Bayesian approaches are "nonsensical" from each other's perspective in more subtle ways.

As Xi'an says, the question is moot. But I think that many people are nevertheless led to consider the maximum-likelihood estimate from a Bayesian perspective because of a statement that appears in some literature and on the internet: that "the maximum-likelihood estimate is a particular case of the Bayesian maximum a posteriori estimate, when the prior distribution is uniform". This statement is incorrect, or needs a lot of qualifications at the very least, and is discussed below. I consider a one-dimensional parameter for simplicity.

First: a Bayesian point of view
It always makes sense to speak of the probability for a continuous parameter, if we are uncertain about it. Ideally we should report our uncertainty by specifying the full probability density for the parameter, conditional on data and other evidence $D$, which we can write as $$p(x \mid D)\,\mathrm{d}x.$$

A point needs to be stressed now: the notation above should remind us that probabilities of continuous quantities always refer to intervals (more precisely, sets) of values of such quantities, never to particular values; although in singular cases we can consider sets containing one value only, for example. The notation above is telling us that

  1. we have chosen a coordinate $x$ on the parameter manifold,
  2. this coordinate allows us to speak of intervals of equal width,
  3. the probability that the unknown parameter value lies in a small interval $\Delta x$ is approximately $p(x \mid D)\,\Delta x$, where $x$ is a point within the interval.
    (Alternatively we can speak of a base Lebesgue measure $\mathrm{d}x$ and intervals of equal measure, but the essence is the same.)

Therefore, a statement like "$p(x_1 \mid D) > p(x_2 \mid D)$" does not mean that the probability at (or for) $x_1$ is larger than that at $x_2$, but that the probability of a small interval around $x_1$ is larger than the probability of an interval of equal width around $x_2$. Such statement is coordinate-dependent.

In some cases it's inconvenient or impossible to specify the full density $p(x \mid D)\,\mathrm{d}x$; therefore we try to informally summarize it. See e.g. Murphy [1]. In extreme cases we summarize it with a single number, an "estimate". An example is the "maximum a posteriori estimate" $\hat{x}$. We could define it as the value at which $p(x \mid D)$ has its (hopefully unique) global maximum: $$ \hat{x} := \arg\max_x p(x \mid D) = \arg\max_x p(D \mid x)\times p(x)\tag{*}\label{*}$$ But more precisely we should say that it's the point such that a small interval around it has greatest probability, compared to the probabilities of other intervals of the same width around other points.

Alright, that was a mouthful, but it makes clear why this estimate is not invariant under reparameterizations or changes of coordinates, $\widehat{f(x)}\neq f(\hat{x})$. Its definition strongly depends on a particular coordinate system (base measure). Informally we could say that this estimate corresponds to dividing the parameter manifold into very small, arbitrary intervals, and then choosing the interval which has maximum posterior probability. Note that this interval division is completely extraneous to the posterior distribution: the latter is coordinate-invariant and independent of any such division. This is also why some authors warns about the dangers of this estimate; see Murphy's [1] and MacKay's [2] discussions for example.

Thus, again informally, we can say that the maximum a posteriori estimate summarizes the posterior distribution not with just one value, but with an implicit division of the parameter space into small intervals, and a choice of a particular interval among them.

Second: the (frequentist) maximum-likelihood point of view
From this point of view, speaking about the probability of a parameter value is simply meaningless. Full stop. We'd like to know what the true parameter value is, and the value $\tilde{x}$ that gives highest probability to the data should intuitively be not too far off the mark: $$\tilde{x} := \arg\max_x p(D \mid x)\tag{**}\label{**}.$$ This is the maximum-likelihood estimate.

This estimate selects a parameter value, not a small interval, and therefore doesn't depend on any coordinate. Stated otherwise: Each point on the parameter manifold is associated with a number: the probability for the data $D$; we're choosing the point that has the highest associated number. This choice does not depend on coordinate systems, or interval divisions, or base measures. It is for this reason that this estimate is parameterization invariant, and this property tells us that it is not a probability – as desired. This invariance remains if we consider more complex reparameterizations, like those that identify points on the parameter manifold, e.g. $x\mapsto x^2$, and the profile likelihood mentioned by Xi'an makes complete sense from this perspective.

Now let's compare the two points of view above:
Comparing eqs $\eqref{*}$ and $\eqref{**}$, we see that they yield the same numerical value if the prior density in the first is constant in $x$: $p(x)=\text{const}$. This is the origin of the statement "the maximum-likelihood estimator is a particular case of the Bayesian maximum a posteriori estimator, when the prior distribution is uniform". But the long discussion above shows – I hope – that the two estimates are in fact completely different. Maximum-a-posteriori is based on an interval division (or coordinate choice, or base measure); maximum-likelihood is not. Maximum-a-posteriori is, in a way, choosing a small interval; maximum-likelihood is choosing a point. The two estimates not only come from different philosophies, but are estimating different things and have completely different properties.

In fact, given any posterior density $$p(x \mid D)\,\mathrm{d}x \propto p(D \mid x)\times p(x)\,\mathrm{d}x,$$ we could always associate a maximum-likelihood estimate with it, thus: (1) reparameterize $y=f(x)$ in such a way that the prior density is uniform in $y$: $p(x)\,\mathrm{d}x \equiv \text{const}\,\mathrm{d}y$; (2) choose the maximum a posteriori estimate in the new parameterization $y$. This again shows that we can't say that one estimate is a particular case of the other.

Is there any other way in which we can consider maximum-likelihood as a particular Bayesian estimate? Let's explore a more rigorous definition of the latter.

Third: a more rigourous Bayesian point of view
It may happen that we must choose one parameter value for some particular purposeor reason, even though the true value is unknown. This choice is the realm of decision theory [3], and the value chosen is the proper definition of "estimator" in Bayesian theory. Decision theory says that we must first introduce a gain function $(x',x)\mapsto G(x'; x)$ which tells us how much we gain by choosing $x'$ when the true parameter value is $x$ (alternatively, we can pessimistically speak of a loss function). This is a two-point function on the parameter manifold, which means that its values are independent of the coordinate system: in a new coordinate system $y=f(x)$, the function has the new expression $(y',y)\mapsto G[f^{-1}(y'); f^{-1}(y)]\equiv G(x'; x)$ [4].

The estimator $\hat{x}$ associated with this gain function is the value $x$ that maximizes the expected gain given our data $D$: $$\hat{x} := \arg\max_{x'} \int G(x'; x)\, p(x \mid D)\,\mathrm{d}x.\tag{***}\label{***}$$ This definition is independent of coordinate changes: in new coordinates $y=f(x)$ the estimator is $\hat{y}=f(\hat{x})$. This follows from the coordinate-independence of $G$ and of the integral.

Now we can ask: is there a gain function that leads to an estimator equal to the maximum-likelihood value? Since the estimator is invariant, unlike the informal estimate previously discussed, we could then really say that maximum-likelihood is a particular case of a Bayesian estimator.

At first it could seem that by choosing $G(x'; x) = \delta(x'-x)$, a Dirac delta, eq. $\eqref{***}$ above would yield $\hat{x} = \arg\max_{x} p(x \mid D)$, which with a uniform prior would be equal to the maximum-likelihood estimate. Jaynes [5] claims this, for example. But a second thought shows that this can't make sense. A Dirac delta returns the value of a function at a specific point; but as we discussed at length above, the probability of a continuous quantity has no value at any specific points. Such values can be formally introduced only if we choose a coordinate $x$ or base measure $\mathrm{d}x$. (More exactly, a Dirac delta is itself a measure, we can't apply it to another measure.)

Maybe we could make sense of the Dirac delta by saying that it implicitly selects a base measure. Then a probability density could be defined, and we could speak about its value at a point (we would be defining a Radon-Nikodym derivative). Such a mathematical object, however, would be no longer a two-point function, but a sort of function relating two intervals. Thus it couldn't be a gain function, because the latter tells us the gain of choosing a point on the parameter manifold when another point is the correct one; it doesn't deal with intervals. Is there an interval-based definition of gain function? Are there other ways out?

Here I stop, because I don't have the slightest idea of whether there's a solution from this point of view. The literature seems to be divided on this, and I refer you to some samples from it [6].

Final comments
The discussion above is sloppy, but can be made precise using measure theory and Stieltjes integration. I apologize for the maybe excessive pedagogical tone; I hope it'll be useful if people who are learning these topics stumble upon this question.

[1] K. P. Murphy: Machine Learning: A Probabilistic Perspective (MIT Press 2012), especially chap. 5.
[2] D. J. C. MackKay: Information Theory, Inference, and Learning Algorithms (Cambridge University Press 2003), http://www.inference.phy.cam.ac.uk/mackay/itila/.
[3] For example, H. Raiffa, R. Schlaifer: Applied Statistical Decision Theory (Wiley 2000).
[4] Y. Choquet-Bruhat, C. DeWitt-Morette, M. Dillard-Bleick: Analysis, Manifolds and Physics. Part I: Basics (Elsevier 1996), or any other good book on differential geometry.
[5] E. T. Jaynes: Probability Theory: The Logic of Science (Cambridge University Press 2003), §13.10.
[6] I. H. Jermyn: Invariant Bayesian estimation on manifolds https://doi.org/10.1214/009053604000001273; R. Bassett, J. Deride: Maximum a posteriori estimators as a limit of Bayes estimators https://doi.org/10.1007/s10107-018-1241-0.

pglpm
  • 1.3k
  • 9
  • 20