When does the maximum likelihood correspond to a reference prior?

Question

I have been reading James V. Stone's very nice books "Bayes' Rule" and "Information Theory". I want to know which sections of the books I did not understand and thus need to re-read further. The following notes which I wrote down seem self-contradictory:

1. The MLE always corresponds to the uniform prior (the MAP of the uniform prior is the MLE).
2. Sometimes a uniform prior is not possible (when the data lacks an upper or lower bound).
3. Non-Bayesian analysis, which uses the MLE instead of the MAP, essentially sidesteps or ignores the issue of modeling prior information and thus always assumes that there is none.
4. Non-informative (also called reference) priors correspond to the maximizing the Kullback-Leibler divergence between posterior and prior, or equivalently the mutual information between the parameter $\theta$ and the random variable $X$.
5. Sometimes the reference prior is not uniform, it can also be a Jeffreys prior instead.
6. Bayesian inference always uses the MAP and non-Bayesian inference always uses the MLE.

Question: Which of the above is wrong?

Even if non-Bayesian analysis does not always correspond to "always use the MLE", does MLE estimation always correspond to a special case of Bayesian inference?

If so, under which circumstances is it a special case (uniform or reference priors)?

Based on the answers to questions [1][2][3][4] on CrossValidated, it seems like 1. above is correct.

The consensus of a previous question I asked seems to be that non-Bayesian analysis cannot be reduced to a special case of Bayesian analysis. Therefore my guess is that 6. above is incorrect.

Answer 1

1. Correct, as long as the support of the uniform prior contains the MLE. The reason for this is that the posterior and the likelihood are proportional on the support of the uniform prior. Even if the MAP and MLE coincide numerically, their interpretation is completely different.
2. False. The support of the prior is certainly location and scale dependent (e.g. if the data are reported in nanometers or in parsecs), but an appropriate choice is often possible. You may need to use a huge compact set as the support, but it is still possible.
3. It does not use prior information in the sense of a prior distribution (since they are completely different inferential approaches) but there is always information injected by the user. The choice of the model is a form of prior information. If you put 10 people to fit a dataset, some of them would probably come up with different answers.
4. Yes. Have a look at the following references

The formal definition of reference priors

Jeffreys Priors and Reference Priors

5. The reference prior and the Jeffreys prior are the same in uniparametric models (unidimensional parameter), but this is not the case in general. They are uniform for location parameters, but this is not the case of scale and shape parameters. They are different even for the scale parameter of the normal distribution (see my previous references).

6. False. Truly Bayesians use the posterior distribution in order to obtain Bayes estimators. The MAP is one of them, but there are many others. See Wikipedia's article on the Bayes estimator.

Non-Bayesians do not always use the MLE. An example of this is the James-Stein estimator, which is based on a different criterion than maximizing a likelihood function.

Answer 2

A few more remarks in addition to the points made by Richard Price:

1. The MLE always corresponds to the uniform prior (the MAP of the uniform prior is the MLE).

This is incorrect for a simple if often overlooked reason: the MLE does not require a dominating measure on the parameter space while a Bayesian approach does. This means that both "the" flat (constant) prior and "the" MAP are actually depending on the choice of the dominating measure. An alternative explanation (already made in a comment) is that the MLE is invariant by reparameterisation, that is under any bijective transform of the parameter, while a flat prior does not remain constant under bijective transforms and the MAP is not invariant by reparameterisation. My general view on MAPs is that they are not Bayesian procedures.

2. Sometimes a uniform prior is not possible (when the data lacks an upper or lower bound).

This is both correct and incorrect. Choosing a Uniform prior $\mathcal U(a,b)$ is always possible, but requires a choice of $a$ and $b$. If the prior density is constant over the entire parameter space (against the chosen dominating measure) then it is not a Uniform density because it is not a probability density. The prior then becomes improper, that is, a $\sigma$-finite measure.

3. Non-Bayesian analysis, which uses the MLE instead of the MAP, essentially sidesteps or ignores the issue of modeling prior information and thus always assumes that there is none.

This is too vague a statement to validate or invalidate. As signaled by Richard Price]1, the choice of a model is a type of information, which may become increasingly Bayesian when bringing in random effects for instance. Further, non-Bayesian analysis is not defined as an approach per se.

4. Non-informative (also called reference) priors correspond to the maximizing the Kullback-Leibler divergence between posterior and prior, or equivalently the mutual information between the parameter 𝜃 and the random variable 𝑋.

Correct: Reference priors in the specific sense of Bernardo (1979), Berger, Bernardo, and Sun (2009), and others, are maximising the expected Kullback-Leibler divergence between prior and posterior for the parameter(s) of interest. As this is usually impossible when considering proper priors, it gets complicated.

5. Sometimes the reference prior is not uniform, it can also be a Jeffreys prior instead.

This is again a vague and not so useful statement. For the same reason as above, namely the lack of invariance under reparameterisation, the reference prior [assuming a specific definition of said prior] is almost never uniform. Jeffreys' approach enjoys invariance under reparameterisation in the sense that its definition is consistent under changes of parameterisation.

6. Bayesian inference always uses the MAP and non-Bayesian inference always uses the MLE.

This is incorrect, for both parts. Bayesian inference always uses the full posterior distribution and only derives procedures like point estimates in cases a decision is required and a loss function provided. MAP estimates are not available as decision-theoretic procedures. Non-Bayesian inference covers all possible answers to an inference problem and hence cannot be characterised.