It has been 2 weeks now I am working on SemEval task 4 (2016) : Sentiment Analysis on Twitter.

The results I achieve are lower than what I expected for the three class classification problem : predict if a tweet is negative, neutral or positive. I tried a bunch of classification techniques, all based on the bag-of-words or tf-idf vectorization. Moreover, I added some custom features such as the number of exclamation marks and positive/negative emoticons. I also tried to add a sentiment score thanks to SentiWordNet lexicon. Other lexicons are planned to be used.

Now the point: all of these hand-engineered features do not increase the score, or really slightly. Even things like lemmatization, stopwords removal, and feature selection up to 90% (with chi2 score) do not affect significantly the score.

Finally, I have a class imbalance and I clearly suppose it to affect the score as the confusion matrix and scores for each class (ordered by negative, neutral, positive on the image) show it :

Scores of classification. Support shows respectively number of negative, neutral and positive tweets

Moreover, when I did undersampling for neutral and positive classes in order to have around the same number of negative samples, accuracy increased by 0.05%.

So, my three questions are :

  1. Anyone has ideas to improve my model ? Ideas of new features, text cleaning, new algorithms (ensembling techniques maybe ?), or completely new approach to the problem than supervised learning ?
  2. How is it possible that only retaining 10% of the vocabulary gives the same scores than the whole ? Is the rest really useless ?
  3. Is there a way to efficiently deal with the imbalance in my case, knowing that apparently undersampling is rarely recommended?

Last thing, I don't have the test data so I evaluate with cross validation on the training set, with stratified folds.

Thank you in advance for any insights.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.