In text categorization, it is common to use tokens as features. But, there are several different ways to convert a sentence to tokens. For example, see the different types of Tokenizers here: https://github.com/NaturalNode/natural . What is the most commonly used tokenizer for categorizing short texts?

  • $\begingroup$ What sort of text are you working with? Is it noisy web text or short snippets from a book? For the latter you could use a simple regular expression... $\endgroup$ – Leeor Sep 16 '13 at 12:24
  • $\begingroup$ I work with chat sentences. They are noisy, with lots of spelling mistakes and grammatical errors. $\endgroup$ – Erel Segal-Halevi Sep 16 '13 at 12:43
  • 1
    $\begingroup$ Well I guess this is more of a question in programming, since you are asking about a tool... I've had fantastic experience with the Stanford NLP tokenizer. Link: nlp.stanford.edu/software/corenlp.shtml $\endgroup$ – Leeor Sep 16 '13 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.