1
$\begingroup$

It has been 2 weeks now I am working on SemEval task 4 (2016) : Sentiment Analysis on Twitter.

The results I achieve are lower than what I expected for the three class classification problem : predict if a tweet is negative, neutral or positive. I tried a bunch of classification techniques, all based on the bag-of-words or tf-idf vectorization. Moreover, I added some custom features such as the number of exclamation marks and positive/negative emoticons. I also tried to add a sentiment score thanks to SentiWordNet lexicon. Other lexicons are planned to be used.

Now the point: all of these hand-engineered features do not increase the score, or really slightly. Even things like lemmatization, stopwords removal, and feature selection up to 90% (with chi2 score) do not affect significantly the score.

Finally, I have a class imbalance and I clearly suppose it to affect the score as the confusion matrix and scores for each class (ordered by negative, neutral, positive on the image) show it :

Scores of classification. Support shows respectively number of negative, neutral and positive tweets

Moreover, when I did undersampling for neutral and positive classes in order to have around the same number of negative samples, accuracy increased by 0.05%.

So, my three questions are :

  1. Anyone has ideas to improve my model ? Ideas of new features, text cleaning, new algorithms (ensembling techniques maybe ?), or completely new approach to the problem than supervised learning ?
  2. How is it possible that only retaining 10% of the vocabulary gives the same scores than the whole ? Is the rest really useless ?
  3. Is there a way to efficiently deal with the imbalance in my case, knowing that apparently undersampling is rarely recommended?

Last thing, I don't have the test data so I evaluate with cross validation on the training set, with stratified folds.

Thank you in advance for any insights.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.