Is there a method to perform tagging of tweets using NLTK? The pos_tag() function gives incorrect results on twitter data (which uses textese):

# checking if NLTK tokenizers work on SMS textese
tokens = pos_tag(word_tokenize("ikr smh he asked fir yo last name so he can
                                add u on fb lololol"))


 [('ikr', 'NN'), ('smh', 'NN'), ('he', 'PRP'), ('asked', 'VBD'), ('fir', 'JJ'), ('yo', 'NN'),
  ('last', 'JJ'), ('name', 'NN'), ('so', 'RB'), ('he', 'PRP'), ('can', 'MD'), ('add', 'VB'),
  ('u', 'JJ'), ('on', 'IN'), ('fb', 'NN'), ('lololol', 'NN')]

2 Answers 2


The ARK Twitter Part-of-Speech Tagger from Prof. Noah Smith's group (back when he was at CMU) is one such resource. The tagset is different from the standard Penn Treebank tagset, and has been created to handle idiosyncrasies of the language on Twitter (http://www.aclweb.org/anthology/P11-2008) Though the library is in Java, I think there's a python wrapper available somewhere out there!


Following is the link for the Python wrapper for CMU tweet tagger,


Its a Simple Python wrapper around runTagger.sh of ark-tweet-nlp. It passes a list of tweets to runTagger.sh and parses the result into a list of lists of tuples, each tuple represents the (token, type, confidence). Download the ark-tweet-nlp jar to use it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.