2
$\begingroup$

In Bayesian theorem, $p(y|x)=\frac{p(y)p(x|y)}{p(x)}$, we call p(y) the prior, p(x|y) the likelihood. While in machine learning, many models find the solution for parameters through the Maximum Likelihood Estimation(MLE) and then apply derivative to find the solutions. I thought the MLE here corresponds to the likelihood in Bayesian function, but it seems not true. For example, in Gaussian discriminative model, the MLE target is $p(t,X|\pi,\mu_1,\Sigma)$, which is a joint distribution. In logistic regression, the MLE target is $p(t|X, w)$, which looks like the "real likelihood" for me. I am confused about this.

  1. The latter MLE is different from the likelihood in Bayesian function, right?
  2. How to tell which kind of MLE to use in machine learning optimization?
$\endgroup$

1 Answer 1

7
$\begingroup$

The $p(x|y)$ in Bayes' theorem is the likelihood function, whereas the MLE is the maximum of this function, which is a point. The likelihood function is not unique for Bayesian methods, it is also used in classical statistics. (Answer for 1.)

I don't have a good answer for part 2). But I would like to point out some pitfalls:

The likelihood function is not a probability distribution, despite its appearance, as it is (almost never) normalized to 1.

To make things worse, the notation for $p(x|y)$ stems from mathematical logic, where the $x$ is investigated under the assumption -the vertical bar- that $y$ remains fixed.

However, the MLE maximizes $p(x|y)$ as a function of $y$, while keeping the $x$ fixed. So the notation for $p(x|y)$ is counterintuitive.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.