2
$\begingroup$

It might be some straight forward thing.. But I referred to some threads already over internet to understand what exact does it mean when we use terms "evidence maximization" "type II maximum likelihood" or "maximum marginal likelihood".

All are same, and work with the marginal likelihood. i.e we maximize the denominator in Bayes Theorem. How does it differ from MLE?

$\endgroup$
1
  • 2
    $\begingroup$ Perhaps you might have a look at the following wiki on empirical Bayes which is the same as type-II maximum likelihood, only the latter name is not listed in wiki: en.wikipedia.org/wiki/Empirical_Bayes_method. $\endgroup$
    – microhaus
    Mar 20, 2021 at 13:10

2 Answers 2

2
$\begingroup$

Empirical Bayes is a means of using the observed data to compute point estimates of the hyperparameters parametrising your priors. Which only makes sense in context of a hierarchical Bayesian model, where you have hyperparameters which parametrise priors on your model parameters.

Maximum likelihood is a frequentist approach - you compute point estimates of the parameters, and there is no uncertainty being modelled in these parameters through the use of priors, parametrised by hyperparameters, on said parameters.

$\endgroup$
2
  • $\begingroup$ Can you say even further that any point estimate is frequentist approach? $\endgroup$
    – Good Luck
    Mar 23, 2021 at 20:53
  • $\begingroup$ @GoodLuck No, that is an incorrect statement. $\endgroup$ Mar 27 at 6:31
0
$\begingroup$

Consider the Latent Variable Model below, and pay particular attention to the plate (the box):

enter image description here

This indicates that the latent Y is not per x, otherwide the plate should have covered the latent Y as well. I use MLE in this case. However, if there is Y per x, and I only have the observed data, but not the corresponding Y for each observed x, then I have no choice but to marginalize out z and maximize p(x), i.e. the evidence (MLE-II).

Summary: solving the model above, was done via maximzing P(x). In words P(x) is called

  • evidence (name stems from Bayes rule)
  • Marginal Likelihood (because it is like P(x|z) but z is marginalized out.
  • Type || MLE ( to distinguish it from standard MLE where you maximize P(x|z).

Almost invariably, you cannot afford to do MLE-II because the evidence is intractable. This is why MLE-I is more common. Even if latent is not given, people still do MLE-I (this is why EM was invented).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.