I have a lot of twitter data (4GB) related to keyword Ebola. I want to classify the tweets into 21 categories.

Categories :-

  1. Death - tweet is about death
  2. Health Care Workers - tweet is about Health Care Workers.
  3. Hospitals and Treatment Facilities
  4. Transmission
  5. Vaccines
  6. Signs/Symptoms

Like these there are 21 categories.

If the categories were less like 4 or 5. I would have used Naive Bayes or SVM. But since tweet text is only 140 characters. I don't know if using Naive Bayes or any other supervised learning techniques is suitable here for so many categories.

I also don't think something like clustering or LDA can give good results.

I can't think of any other way to move forward as categories are not very dissimilar and some overlapping of meaning is there.

So, the question is how do I classify tweets with so many categories? Which method will be more applicable here?

  • 2
    $\begingroup$ What's actually the question? $\endgroup$ – Firebug Jul 20 '16 at 23:00
  • $\begingroup$ I've edited the question $\endgroup$ – Nick Jul 21 '16 at 0:06
  • $\begingroup$ Are all categories binary, and do you have labels for all 4GB tweets? $\endgroup$ – Franck Dernoncourt Sep 6 '16 at 17:30

1) One method is to reduce the complexity of the problem. Instead of training a classifier for 21 classes, you can train 21 binary classifiers. You pick the final class based on the softmax of the predictions of the 21 classifiers.

2) Another supervised learning technique is neural networks. Your output layer will have a size of 21. Your input can be a sparse input as with SVM, or you can use word embeddings (e.g. word2vec, glove) to create a dense embedding of the tweet.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.