Why don't we estimate the prior in a Naive Bayes' classifier?

Question

I'm currently studying the textbook Introduction to Machine Learning 4e (Ethem Alpaydin) the brush up on my ML basics and had a question regarding a part w.r.t. using the Naive Bayes' classifier in multivariate analysis. More specifically, this is the part that's confusing me:

Let us say $x_j$ are binary where $p_{i, j} = p(x_j = 1\ \vert\ C_i)$. If $x_j$ are independent binary variables, we have $$p(\mathbf{x}\ \vert\ C_i) = \prod_{j = 1}^d p_{i, j}^{x_j} (1 - p_{i, j})^{(1 - x_j)}$$ This is another example of the naive Bayes' classifier where $p(x_j\ \vert\ C_i)$ are Bernoulli. The discriminant function is: $$ \begin{align} g_i(\mathbf{x}) & = \log{(p(\mathbf{x}\ \vert \ C_i))} + \log{(P(C_i))} \\ & = \sum_j \left[ x_j \log{(p_{i, j}) + (1 - x_j) \log{(1 - p_{i, j})}} \right] + \log{(P(C_i))} \end{align} $$ which is linear. The estimator for $p_{i, j}$ is: $$\hat{p}_{i, j} = \frac{\sum_t x_j^t r_i^t}{\sum_j r_i^t}$$ ($r_i^t = 1$ if $\mathbf{x}^t \in C_i$).

What's confusing me is, I recall in an earlier chapter about the Bayes' classifier and parametric classification that we may also use maximum likelihood estimation (MLE) to get the estimate for the prior $P(C_i)$ such that

$$ \hat{P}(C_i) = \frac{\sum_t r_i^t}{N} $$

Why is it that estimation isn't made here? I thought that it was implied, but it seems to be omitted altogether.

Answer 1

The difference is that for navie Bayes you are calculating conditional probability $p(x_j|C_i)$, so you need to calculate the probability that $X=x_j$ given that $C_i=1$. With binary $r_i$ variables you count only such events (multiply by zeros and ones) and divide by number of such cases (sum ones). The conditional probability $p(x_j|C_i)$ together with marginal probability $p(C_i)$ calculated as usual will then enable you to use Byes theorem and get the opposite conditional probability

$$ p(C_i|x_j) \propto p(x_j|C_i)\, p(C_i) $$

You want to know this, so that you can make predictions: if I observed $x_j$, than I predict that the class is probably $C_i$.

Answer 2

Wikipedia on Naive Bayes says "A class's prior may be calculated by assuming equiprobable classes (i.e., priors = $1/K$ for $K$ classes), or by calculating an estimate for the class probability from the training set (i.e., priors = $N_k/N$ for $N$ total obs; $N_k$ obs with label $k$.)." It seems like you're expecting the second method (learned) and the book is showing you the first method (uniform). If you expect the training and test sets to be drawn from the same joint distribution, I agree it seems better to update the class probabilities based on the training data.