While implementing a Naive Bayes filter, I stumbled across a problem with the calculation of the conditional probabilities $p(w|c)$ of a word $w \in \mathcal{W}$ given a class $c \in \mathcal{C}$. Following this answer on StackOverflow about classifying fruits $\mathcal{C} = \left\{ \text{banana}, \text{orange}, \text{other} \right\}$ having the features $\mathcal{W} = \left\{\text{long}, \text{sweet}, \text{yellow}\right\}$, this is what my problem is about:

Assume we have the three observed sentences (or documents) $\mathcal{D} = \left\{ D_{1}, \dotsc, D_{3}\right\}$ in the class $c = \text{banana}$ as

$ \begin{align} D_1 = \left\{\text{long}, \text{sweet}, \text{yellow}\right\} &\to \left\{\text{long}, \text{sweet}, \text{yellow}\right\} \\ D_2 = \left\{\text{long} \right\} &\to \left\{\text{long}, \neg \text{sweet}, \neg \text{yellow} \right\} \\ D_3 = \left\{ \text{yellow}, \text{sweet} \right\} &\to \left\{ \neg \text{long}, \text{yellow}, \text{sweet} \right\} \end{align} $

then what is the probability $p(w = \text{long}|c = \text{banana})$?

According to the linked answer, the probability would be calculated as

$ \begin{align} p(w = \text{long}|\text{banana}) &= \frac{\left|w\right|}{\left|\mathcal{D}\right|} = \frac{\left|w\right|}{\left|w\right| + \left|\neg w\right|} = \frac{2}{3} = 0.\overline{6} \end{align} $

because the word appears in two out of three sentences. Regarding the fact that I later on want to classify documents/sentences of different lengths (think mail classification), can't we also say, that the probabilities of the word appearing in the individual training sentences are:

$ \begin{align} p(w = \text{long}|\text{D}_1) &= \frac{\left|w\right|}{\left|\text{D}_1\right|} = \frac{1}{3} = 0.\overline{3} \\ p(w = \text{long}|\text{D}_2) &= \frac{\left|w\right|}{\left|\text{D}_2\right|} = \frac{1}{1} = 1.0 \\ p(w = \text{long}|\text{D}_3) &= \frac{\left|w\right|}{\left|\text{D}_3\right|} = \frac{0}{2} = 0.0 \end{align} $

because the word appears $1$ out of $3$ times in $\text{D}_1$, etc. Then wouldn't the (well, expected) probability be

$ \begin{align} p(\text{long}|\text{banana}) &= \frac{1}{N} \sum_{i=1}^{N} p(\text{long}|\text{D}_N) = \frac{1}{3}\left(\frac{1}{3} + \frac{1}{1} + \frac{0}{2}\right) = \frac{4}{9} = 0.\overline{4} \end{align} $ (?)

which looks like the complement by accident.

Or is the way to think that

$ \begin{align} p(\text{long}|D_1) &= p(\text{long}|\text{long}, \text{sweet}, \text{yellow}) = p(\text{long}|\text{long}) = 1 \\ p(\text{long}|D_2) &= p(\text{long}|\text{long}, \neg \text{sweet}, \neg \text{yellow}) = p(\text{long}|\text{long}) = 1 \\ p(\text{long}|D_3) &= p(\text{long}|\neg \text{long}, \text{sweet}, \text{yellow}) = p(\text{long}|\neg \text{long}) = 0 \end{align} $

because $\text{long}$ is conditionally independent of all other features (following the Naive assumption)? But then, what about double-counting words (i.e. in mail classification)?

The approach of counting over all words would yield $2/6 = 0.\overline{3}$ here, which is another different number. I assume that averaging combination is not the way to go, but how does the combination of the per-sentence probabilities work right?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.