2
$\begingroup$

I was looking at the Naive Bayes classifier models (Binomial, Multinomial and Gaussian) and trying to understand the theory behind them a bit better, but am unsure if I understand the MLE approach correctly and use the correct terminology.

Let's use C for Class and W for word.

Then the Bayes Theorem looks like this: $p(C|W) = \frac{p(W|C)*p(C)}{p(W)}$.

We ignore the denominator and take the log of everything:

$ln(p(C|W)) = ln(p(W|C)) + ln(p(C))$

So, this part $p(W|C)$ is called the likelihood and $ln(p(W|C))$ is then the log-likelihood. This $ln(p(C))$ is the log class prior.

The entire expression is the log posterior.

And even though only one part of the formula is called likelihood, we use the Maximum Likelihood approach both for the likelihood and the prior, right?

In a two-class Naive Bayes, the prior is a Binomial, so we just do an MLE for that one parameter. In a multi-class-problem, it's a Multinomial.

And if the class-conditional is, say, a Gaussian, we do the MLE for the two parameters of that Gaussian.

What I am getting at: It's a bit confusing that only one part of the formula is called likelihood but we actually do two separate ML estimations, one for the actual likelihood and one for the prior.

Is there anything wrong with the above?

One reason I am asking this is because I was looking at this blog post https://towardsdatascience.com/quadratic-discriminant-analysis-ae55d8a8148a and under "Derivation and training", I think the writer uses the term "log-likelihood" for what I called "log posterior distribution".

Help and feedback are appreciated!

Improve this question
$\endgroup$
3
  • $\begingroup$ A maximum likelihood approach (MLE) takes the $C$ which maximises $p(W \mid C)$ or equivalently $\log(p(W \mid C))$; this is not a Bayesian approach. A maximum a posteriori probability approach (MAP) takes the $C$ which maximises $p(W \mid C)p(C)$ or equivalently $\log(p(W \mid C))+\log(p(C))$, i.e. combined not separately; this has Bayesian elements but is not really a Bayesian approach, which would instead use the posterior distribution to minimise a loss function. But in answer to your question, they are different approaches $\endgroup$
    – Henry
    Sep 18, 2022 at 12:33
  • $\begingroup$ Naive Bayes is another issue which uses Bayesian elements but is not really Bayesian: it makes assumptions about conditional independence without obvious justification $\endgroup$
    – Henry
    Sep 18, 2022 at 12:37
  • $\begingroup$ @Henry I don't understand your comment: there is a p(C) in Naive Bayes even for the MLE approach. $\endgroup$
    – user3629892
    Sep 18, 2022 at 12:49

1 Answer 1

Reset to default
4
$\begingroup$

The parameters of naive Bayes are the $p(W|C)$ and $p(C)$ probabilities. You estimate them using maximum likelihood and plug in into the Bayes theorem to calculate the $p(C|W)$ probabilities. The model is not Bayesian because it doesn't use subjective probabilities, you are not using it to update the prior. “Likelihood” is used here in two meanings: as likelihood in MLE and as the name of the component of the Bayes theorem, which made it confusing.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Ok, so just to be clear: the likelihood in MLE is $p(W|C)p(C)$ and it in the case of two-class Gaussian NB contains 3 parameters: $\mu$, $\sigma$ and $\pi$ $\endgroup$
    – user3629892
    Sep 18, 2022 at 12:50
  • 1
    $\begingroup$ @user3629892 no $p(W|C)$ and $p(C)$ are the parameters here, with Gaussian NB you got it correctly. Bayes theorem is here just a “computation” on the parameters, that each is estimated using MLE. It's not the likelihood, you estimate each of the parameters independently. $\endgroup$
    – Tim ♦
    Sep 18, 2022 at 12:54
  • $\begingroup$ hm.. I don't get it: in the case of Gaussian NB, $p(W|C)$ is a Gaussian distribution. And that distribution has two parameters, $\mu$ and $\sigma$. And I estimate these two parameters using MLE. And $p(C)$ is a binomial, with one parameter $\pi$ and I estimate that also using MLE. Or do you mean, that the entire Gaussian distribution is one of the parameters? $\endgroup$
    – user3629892
    Sep 18, 2022 at 13:32
  • 1
    $\begingroup$ @user3629892 yes, in Gaussian NB you estimate the parameters of the Gaussian, but the probability density returned by Gaussian is a parameter of naive Bayes. $\endgroup$
    – Tim ♦
    Sep 18, 2022 at 14:02
  • 1
    $\begingroup$ @user3629892 correct. $\endgroup$
    – Tim ♦
    Sep 18, 2022 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.