I am working with a semantic analysis problem and wanted to know if anyone has been able to set a default value, say a probability of zero or 0.5 for phrases/words that the machine learning algorithm has never seen. Using scikit-learn's classifiers and nltk's word_vectorizer I have experienced probability predictions of 1.0 for words and phrases not in the training, which is a potentially misleading output with absolute confidence.

Would adding a dictionary of English not in the training with a target of zero help?

What about non-words or incorrect spellings? How do you punish the unknown/unseen without explicitly punishing all permutations of words or word-spellings not in the corpus?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.