2
$\begingroup$

Using Laplacian Smoothing we can get rid of 0 probabilities if a term occur in spam and does not occur in ham class or vice versa. My question is about what if a term in test document does not occur in training dataset(i.e. in dictionary). For example if we extend example in page 44 in -> http://www.stanford.edu/class/cs124/lec/naivebayes.pdf as follows:

(Consider only change is test document part)

Test 5 Chinese Chinese Chinese Tokyo Japan Greek ?

P(c|d5)'s calculation requires P(Greek|c). However since Greek doesn't exist in dictionary we didn't calculate it before. What should P(Greek|c)'s and P(Greek|j)'s value?

$\endgroup$

2 Answers 2

2
$\begingroup$

One common solution is to treat tokens seen less than $n$ times (across all classes) as a special "unknown" or "rare" token. You then use this probability to assign values to legitimately unknown known words.

You certainly don't want to assign zero probability to all classes when you encounter a novel word. This avoids that issue, and captures the idea that different classes may have different amounts of out-of-dictionary words.

$\endgroup$
0
$\begingroup$

One solution is to map all of the unknown words to some token like . Then you can assign some probability to that token for each of your classes.

$\endgroup$
6
  • $\begingroup$ As I understand, value does not matter. However, for example If a test document has a lot of unknown terms, high value for unknown terms affect results as reward; low value for unknown terms affect results as penalty. Doesn't it? Can proof it by using equations? $\endgroup$ Commented May 16, 2014 at 20:15
  • $\begingroup$ I'm not sure what you mean by high value and by reward. $\endgroup$
    – Aaron
    Commented May 16, 2014 at 20:25
  • $\begingroup$ If assigned probability value to the unknown words(which is same for all of them as you said) is high, P(c|d5) will be high; otherwise P(c|d5) will be low. I meant in my first comment, is this fair when we compare documents which contain any non-dictionary terms with documents which contain a lot of non-dictionary terms? $\endgroup$ Commented May 17, 2014 at 5:08
  • $\begingroup$ That's the whole point. If unknown words are associated more with class c than with class j then you would want P(c|d5) to be high when the document has unknown words. You have to learn the probability of unknown words for each class using held-out data. $\endgroup$
    – Aaron
    Commented May 17, 2014 at 6:39
  • $\begingroup$ I understood from the message that i copied link, P(Greek|c) and P(Greek|j) must be very low probaility, for example 0.00000001. But in here link they simply ignored missing words. Also you said I have to learn unknown words probability using held-out(?) data. What is held-out data? Is it training set? That was the problem, we do not have unknown words in it. I'm very confused. $\endgroup$ Commented May 17, 2014 at 7:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.