6
$\begingroup$

I have been reading James V. Stone's very nice books "Bayes' Rule" and "Information Theory". I want to know which sections of the books I did not understand and thus need to re-read further. The following notes which I wrote down seem self-contradictory:

  1. The MLE always corresponds to the uniform prior (the MAP of the uniform prior is the MLE).
  2. Sometimes a uniform prior is not possible (when the data lacks an upper or lower bound).
  3. Non-Bayesian analysis, which uses the MLE instead of the MAP, essentially sidesteps or ignores the issue of modeling prior information and thus always assumes that there is none.
  4. Non-informative (also called reference) priors correspond to the maximizing the Kullback-Leibler divergence between posterior and prior, or equivalently the mutual information between the parameter $\theta$ and the random variable $X$.
  5. Sometimes the reference prior is not uniform, it can also be a Jeffreys prior instead.
  6. Bayesian inference always uses the MAP and non-Bayesian inference always uses the MLE.

Question: Which of the above is wrong?

Even if non-Bayesian analysis does not always correspond to "always use the MLE", does MLE estimation always correspond to a special case of Bayesian inference?

If so, under which circumstances is it a special case (uniform or reference priors)?

Based on the answers to questions [1][2][3][4] on CrossValidated, it seems like 1. above is correct.

The consensus of a previous question I asked seems to be that non-Bayesian analysis cannot be reduced to a special case of Bayesian analysis. Therefore my guess is that 6. above is incorrect.

$\endgroup$
  • 1
    $\begingroup$ Re the title: The likelihood doesn't respond to a prior at all. The posterior does respond to changes in the prior of course. $\endgroup$ – Glen_b Aug 20 '16 at 1:46
  • 1
    $\begingroup$ In 2, the uniformity relates to the parameter (or perhaps function of parameters) you're performing inference on, not the support of the data. In 3. Frequentist inference can bring in outside information (otherwise how would meta-analysis work?); it doesn't incorporate subjective priors but previous data (or its likelihood, if that's all that's available, or even just an approximation to its likelihood obtained from estimates and standard errors) can be readily incorporated. In 5. reference priors needn't be any of the things you mentioned - e.g. in mutivariate settings. 6- MAP is atypical $\endgroup$ – Glen_b Aug 20 '16 at 1:49
7
$\begingroup$
  1. Correct, as long as the support of the uniform prior contains the MLE. The reason for this is that the posterior and the likelihood are proportional on the support of the uniform prior. Even if the MAP and MLE coincide numerically, their interpretation is completely different.
  2. False. The support of the prior is certainly location and scale dependent (e.g. if the data are reported in nanometers or in parsecs), but an appropriate choice is often possible. You may need to use a huge compact set as the support, but it is still possible.
  3. It does not use prior information in the sense of a prior distribution (since they are completely different inferential approaches) but there is always information injected by the user. The choice of the model is a form of prior information. If you put 10 people to fit a dataset, some of them would probably come up with different answers.
  4. Yes. Have a look at the following references

The formal definition of reference priors

Jeffreys Priors and Reference Priors

  1. The reference prior and the Jeffreys prior are the same in uniparametric models (unidimensional parameter), but this is not the case in general. They are uniform for location parameters, but this is not the case of scale and shape parameters. They are different even for the scale parameter of the normal distribution (see my previous references).

  2. False. Truly Bayesians use the posterior distribution in order to obtain Bayes estimators. The MAP is one of them, but there are many others. See Wikipedia's article on the Bayes estimator.

Non-Bayesians do not always use the MLE. An example of this is the James-Stein estimator, which is based on a different criterion than maximizing a likelihood function.

$\endgroup$
  • 1
    $\begingroup$ @William No worries at all, I am just a bird of passage. If you find the answer useful, that's enough for me. $\endgroup$ – Richard Price Aug 20 '16 at 1:15
  • 2
    $\begingroup$ (1) is not true without some caveat. The MLE does not depend on the parameterization. But you cannot specify a "uniform prior" without giving a parameterization. For example, if I put a flat prior on $\sigma$ in a linear model, sure, the MAP estimator of $\sigma$ is also the MLE. But the MAP estimator of $\sigma^2$ is not the MLE of $\sigma^2$ under this prior. $\endgroup$ – guy Aug 20 '16 at 2:01
  • 2
    $\begingroup$ @guy Indeed, the uniform prior has to be chosen under the parametrization of interest. $\endgroup$ – Richard Price Aug 20 '16 at 2:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.