Tagging of tweets using NLTK

Question

Is there a method to perform tagging of tweets using NLTK? The pos_tag() function gives incorrect results on twitter data (which uses textese):

# checking if NLTK tokenizers work on SMS textese
tokens = pos_tag(word_tokenize("ikr smh he asked fir yo last name so he can
                                add u on fb lololol"))

print(tokens)

output:-
 [('ikr', 'NN'), ('smh', 'NN'), ('he', 'PRP'), ('asked', 'VBD'), ('fir', 'JJ'), ('yo', 'NN'),
  ('last', 'JJ'), ('name', 'NN'), ('so', 'RB'), ('he', 'PRP'), ('can', 'MD'), ('add', 'VB'),
  ('u', 'JJ'), ('on', 'IN'), ('fb', 'NN'), ('lololol', 'NN')]

Answer 1

The ARK Twitter Part-of-Speech Tagger from Prof. Noah Smith's group (back when he was at CMU) is one such resource. The tagset is different from the standard Penn Treebank tagset, and has been created to handle idiosyncrasies of the language on Twitter (http://www.aclweb.org/anthology/P11-2008) Though the library is in Java, I think there's a python wrapper available somewhere out there!

Answer 2

Following is the link for the Python wrapper for CMU tweet tagger,

https://github.com/ianozsvald/ark-tweet-nlp-python

Its a Simple Python wrapper around runTagger.sh of ark-tweet-nlp. It passes a list of tweets to runTagger.sh and parses the result into a list of lists of tuples, each tuple represents the (token, type, confidence). Download the ark-tweet-nlp jar to use it.