2
$\begingroup$

Using Laplacian Smoothing we can get rid of 0 probabilities if a term occur in spam and does not occur in ham class or vice versa. My question is about what if a term in test document does not occur in training dataset(i.e. in dictionary). For example if we extend example in page 44 in -> http://www.stanford.edu/class/cs124/lec/naivebayes.pdf as follows:

(Consider only change is test document part)

Test 5 Chinese Chinese Chinese Tokyo Japan Greek ?

P(c|d5)'s calculation requires P(Greek|c). However since Greek doesn't exist in dictionary we didn't calculate it before. What should P(Greek|c)'s and P(Greek|j)'s value?

$\endgroup$
2
$\begingroup$

One common solution is to treat tokens seen less than $n$ times (across all classes) as a special "unknown" or "rare" token. You then use this probability to assign values to legitimately unknown known words.

You certainly don't want to assign zero probability to all classes when you encounter a novel word. This avoids that issue, and captures the idea that different classes may have different amounts of out-of-dictionary words.

$\endgroup$
0
$\begingroup$

One solution is to map all of the unknown words to some token like . Then you can assign some probability to that token for each of your classes.

$\endgroup$
  • $\begingroup$ As I understand, value does not matter. However, for example If a test document has a lot of unknown terms, high value for unknown terms affect results as reward; low value for unknown terms affect results as penalty. Doesn't it? Can proof it by using equations? $\endgroup$ – γιάννης May 16 '14 at 20:15
  • $\begingroup$ I'm not sure what you mean by high value and by reward. $\endgroup$ – Aaron May 16 '14 at 20:25
  • $\begingroup$ If assigned probability value to the unknown words(which is same for all of them as you said) is high, P(c|d5) will be high; otherwise P(c|d5) will be low. I meant in my first comment, is this fair when we compare documents which contain any non-dictionary terms with documents which contain a lot of non-dictionary terms? $\endgroup$ – γιάννης May 17 '14 at 5:08
  • $\begingroup$ Similar to my question, but the reason is not clear, discussion in -> stackoverflow.com/questions/8680243/… In accepted answer in case 5 and in explanation says: "First, if the word has never been observed in some set, the probability of the message in it should decrease. Ignoring the word is not a good idea, you need to account for it as a highly unprobable one." but why? $\endgroup$ – γιάννης May 17 '14 at 5:39
  • $\begingroup$ That's the whole point. If unknown words are associated more with class c than with class j then you would want P(c|d5) to be high when the document has unknown words. You have to learn the probability of unknown words for each class using held-out data. $\endgroup$ – Aaron May 17 '14 at 6:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.