0
$\begingroup$

I am following CS229 course by Andrew Ng. On this lecture note it talks about using Laplace smoothing to bypass situations of 0-probabilities. What does not make sense is the immediate jump to the following line. enter image description here

I cannot understand how this derivation is made. Can anyone show me the full derivation of how the factor of 2 comes into play? intuitively I feel that 2 is because it is a binary classifier, but a more mathematical reasoning is appreciated.

$\endgroup$
1
$\begingroup$

To avoid $\frac{0}{0}$ situation, we add one to each observed events, this is also called psuedo-counting . Since you increase the count of all possible events by 1, you have to increase the total number of trials by the number of possible events.

In binary Naive Bayes, since we increase each event(item being from 0 or 1 ) by 1 you have to change denominator to $N + 1 \times 2$.

In general, we denote $\alpha > 0$ as smoothing(psuedocounting) factor. THen your smoothed probability becomes, \begin{equation} Pr_{smoothed}(y=i | x) = \frac{1_{y=i} + \alpha}{ N + \alpha \times d} \end{equation} where $d$ is number of possible outcomes, and $i$ is a possible outcome out of $d$ outcomes. Hope this helps

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.