2
$\begingroup$

This is from https://scikit-learn.org/stable/modules/naive_bayes.html

In the last line it says

and we can use Maximum A Posteriori (MAP) estimation to estimate $P(y)$ and $P(x_i|y)$; the former is then the relative frequency of class $y$ in the training set.

Why would we estimate $P(y)$ and $P(x_i|y)$ through MAP? Shouldn't we calculate $P(y)$ and $P(x_i|y)$ through training data? enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

There's no such a thing as estimating parameters "through training data". To get point estimates, you can train a probabilistic model by maximizing the likelihood function (MLE) alone, or maximizing the posterior probabilities (MAP). The above description says that it uses MAP, so it considers not only the data but also a prior. In naive Bayes, the common choice is to use Laplace smoothing (uniform prior) to prevent probabilities of zeroes for the unobserved cases, which would zero-out everything in the calculations.

You can just use maximum likelihood estimator, i.e. calculate probability as number of occurrences / sample size, but there are two potential problems:

  1. For every $(x_i, y)$ pair you would need to observe enough data to be able to calculate the $P(x_i|y)$ probability reliably. If $x_i$ is rare or is rare for the particular subgroup $y=c$, you would need to collect a lot of data to calculate it reliably. This is why we often use procedures as Laplace smoothing, or assume other priors so that we can correct the empirical estimates. With Laplace smoothing, we can make the empirical probabilities smoother (more uniform-ish) or more extreme. If you have reasonable prior information to peak an informative prior, you can use it to "correct" the information contained with data based on other information or prior results.

  2. If for some $(x_i, y)$ pair the number of occurrences of $x_i|y$ would be equal to zero, then the estimated probability $P(x_i|y)$ would be also equal to zero 0 / sample size = 0. In such case

    $$ P(y) \prod_{i=1}^n P(x_i | y) $$

    would also zero-out no matter what the other probabilities are. Laplace smoothing fixes this problem by forcing the probabilities to be non-zero.

Using the Bayesian approach (MAP) solves the two problems at the cost of making the estimates of the individual probabilities slightly biased.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I am sorry, I don't quite understand... Why can't we estimate parameters through training data? Say we are building a email spam classifier using Naive Bayes. We have training data as 10000 emails. Now we pick all the spam mails (which is variable y) and calculate how many times a word (which is variable 𝑥𝑖) is appearing in those mails. This would give us the probability 𝑃(𝑥𝑖|𝑦). Where am I going wrong? $\endgroup$
    – Esha
    Commented Aug 21, 2021 at 10:07
  • 1
    $\begingroup$ @Esha this is maximum likelihood. In the documentation, they are saying that they use MAP so consider also priors. $\endgroup$
    – Tim
    Commented Aug 21, 2021 at 10:15
  • 1
    $\begingroup$ Laplace smoothing for word counts may not be your best bet - see What’s wrong with adding one? (Gale and Church, 1994) $\endgroup$
    – Arya McCarthy
    Commented Aug 21, 2021 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.