4
$\begingroup$

With regards to the Naive Bayes classificator, I have read the following in Wikipedia and wanted to know why it is like that:

"In many practical applications, parameter estimation for naive Bayes models uses the method of maximum likelihood; in other words, one can work with the naive Bayes model without accepting Bayesian probability or using any Bayesian methods."

Therefore, if this is the case, I will be ignoring the probability that goes in the denominator:

P(A|B) = P(B|A)*P(A)

Can someone explain me in simple words what it means, please?

$\endgroup$

2 Answers 2

9
$\begingroup$

Bayesian model is defined in terms of likelihood function (probability of observing the data given the parameters) and priors (assumed distributions for the estimated parameters). Naive Bayes algorithm estimates the probabilities directly from the data, so it does not make any assumptions about their distributions (does not use priors), so it is not Bayesian.

The algorithm estimates the joint probability for the target variable $y$ and the features $x_1, x_2, \dots, x_m$ and then classifies by choosing classes that got assigned the greatest probabilities. It is called naive because it makes the "naive" assumption that the features are independent, so we can write

$$ p(x_1, x_2, \dots, x_m \mid y) = \prod_{j=1}^m p(x_j \mid y) $$

and then, using the properties of conditional probability, given this, we can estimate the joint probability

$$ p(y, x_1, x_2, \dots, x_m) = p(x_1, x_2, \dots, x_m \mid y) \; p(y) $$

This may not sound like a big deal, but estimating the probabilities for pairs of variables in $p(x_j \mid y)$ is way easier than estimating the joint probability $p(x_1, x_2, \dots, x_m \mid y)$ all at once.

What we need to estimate here, are the conditional $p(x_j \mid y)$ and marginal $p(y)$ probabilities, and we use maximum likelihood for this. It is nicely explained on this blog and in the The Naive Bayes Model, Maximum-Likelihood Estimation, and the EM Algorithm paper by Michael Collins. For a general introduction to maximum likelihood estimation, check the Maximum Likelihood Estimation (MLE) in layman terms thread.

If you are dealing with binary, or categorical variables, then they follow Bernoulli and categorical distributions, and the maximum likelihood estimators for parameters of those distributions (the probabilities), are simply the empirical proportions. For example, if you observed the sequence of coin tosses $z_1,z_2,\dots,z_n$ that are independent and identically distributed according to Bernoulli distribution with unknown probability of success $\theta$, and among those tosses you observed $k$ heads, then the "most likely" probability of observing heads given this data is $\hat \theta = \tfrac{k}{n}$. So if you previously observed that a third of tosses were heads, then your best guess is, that also third of the future guesses would be heads. Here you can find the formal derivation. The same kind of computations are done for all the probabilities of interest and then everything is multiplied (see formulas above) to obtain the final result.

All this said, if you want, you can estimate the naive Bayes algorithm in the Bayesian paradigm by assuming priors for the probabilities and maximizing the posterior probability rather than the likelihood.

You can also find a detailed explanation of the naive Bayes algorithm with multiple examples on StackOverflow.

$\endgroup$
2
$\begingroup$

Ignoring the denominator is related to posterior maximization, i.e. choosing the class with the highest posterior probability. The quote is actually referring to Bayesian and Frequentist methods. Frequentist methodology uses data to explain phenomenon (i.e. likelihood), while Bayesian methodology assumes priors, other than the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.