1
$\begingroup$

I am working with a semantic analysis problem and wanted to know if anyone has been able to set a default value, say a probability of zero or 0.5 for phrases/words that the machine learning algorithm has never seen. Using scikit-learn's classifiers and nltk's word_vectorizer I have experienced probability predictions of 1.0 for words and phrases not in the training, which is a potentially misleading output with absolute confidence.

Would adding a dictionary of English not in the training with a target of zero help?

What about non-words or incorrect spellings? How do you punish the unknown/unseen without explicitly punishing all permutations of words or word-spellings not in the corpus?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.