3
$\begingroup$

I'm currently studying the textbook Introduction to Machine Learning 4e (Ethem Alpaydin) the brush up on my ML basics and had a question regarding a part w.r.t. using the Naive Bayes' classifier in multivariate analysis. More specifically, this is the part that's confusing me:

Let us say $x_j$ are binary where $p_{i, j} = p(x_j = 1\ \vert\ C_i)$. If $x_j$ are independent binary variables, we have $$p(\mathbf{x}\ \vert\ C_i) = \prod_{j = 1}^d p_{i, j}^{x_j} (1 - p_{i, j})^{(1 - x_j)}$$ This is another example of the naive Bayes' classifier where $p(x_j\ \vert\ C_i)$ are Bernoulli. The discriminant function is: $$ \begin{align} g_i(\mathbf{x}) & = \log{(p(\mathbf{x}\ \vert \ C_i))} + \log{(P(C_i))} \\ & = \sum_j \left[ x_j \log{(p_{i, j}) + (1 - x_j) \log{(1 - p_{i, j})}} \right] + \log{(P(C_i))} \end{align} $$ which is linear. The estimator for $p_{i, j}$ is: $$\hat{p}_{i, j} = \frac{\sum_t x_j^t r_i^t}{\sum_j r_i^t}$$ ($r_i^t = 1$ if $\mathbf{x}^t \in C_i$).

What's confusing me is, I recall in an earlier chapter about the Bayes' classifier and parametric classification that we may also use maximum likelihood estimation (MLE) to get the estimate for the prior $P(C_i)$ such that

$$ \hat{P}(C_i) = \frac{\sum_t r_i^t}{N} $$

Why is it that estimation isn't made here? I thought that it was implied, but it seems to be omitted altogether.

$\endgroup$
3
$\begingroup$

The difference is that for navie Bayes you are calculating conditional probability $p(x_j|C_i)$, so you need to calculate the probability that $X=x_j$ given that $C_i=1$. With binary $r_i$ variables you count only such events (multiply by zeros and ones) and divide by number of such cases (sum ones). The conditional probability $p(x_j|C_i)$ together with marginal probability $p(C_i)$ calculated as usual will then enable you to use Byes theorem and get the opposite conditional probability

$$ p(C_i|x_j) \propto p(x_j|C_i)\, p(C_i) $$

You want to know this, so that you can make predictions: if I observed $x_j$, than I predict that the class is probably $C_i$.

$\endgroup$
4
  • $\begingroup$ So if I understand correctly, the reason why we're not estimating $P(C_i)$ in the case that I posted is because the naive Bayes' classifier we don't need to calculate $P(C_i)$? $\endgroup$ – Seankala Sep 13 '20 at 7:55
  • $\begingroup$ @Seankala you need both to apply Bayes theorem. $\endgroup$ – Tim Sep 13 '20 at 7:59
  • $\begingroup$ Yes, but I'm just confused because in other parts of the chapter where naive Bayes' is discussed, the author wrote $\hat{P}(C_i)$, but in this particular section he didn't. I was just wondering what the reason may be. $\endgroup$ – Seankala Sep 13 '20 at 8:01
  • $\begingroup$ @Seankala it’s impossible to comment without access to actual text. $\endgroup$ – Tim Sep 13 '20 at 8:22
1
$\begingroup$

Wikipedia on Naive Bayes says "A class's prior may be calculated by assuming equiprobable classes (i.e., priors = $1/K$ for $K$ classes), or by calculating an estimate for the class probability from the training set (i.e., priors = $N_k/N$ for $N$ total obs; $N_k$ obs with label $k$.)." It seems like you're expecting the second method (learned) and the book is showing you the first method (uniform). If you expect the training and test sets to be drawn from the same joint distribution, I agree it seems better to update the class probabilities based on the training data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.