It has been 2 weeks now I am working on SemEval task 4 (2016) : Sentiment Analysis on Twitter.

The results I achieve are lower than what I expected for the three class classification problem : predict if a tweet is negative, neutral or positive. I tried a bunch of classification techniques, all based on the bag-of-words or tf-idf vectorization. Moreover, I added some custom features such as the number of exclamation marks and positive/negative emoticons. I also tried to add a sentiment score thanks to SentiWordNet lexicon. Other lexicons are planned to be used.

Now the point: all of these hand-engineered features do not increase the score, or really slightly. Even things like lemmatization, stopwords removal, and feature selection up to 90% (with chi2 score) do not affect significantly the score.

Finally, I have a class imbalance and I clearly suppose it to affect the score as the confusion matrix and scores for each class (ordered by negative, neutral, positive on the image) show it :

Moreover, when I did undersampling for neutral and positive classes in order to have around the same number of negative samples, accuracy increased by 0.5%.

So, my three questions are :

1. Anyone has ideas to improve my model ? Ideas of new features, text cleaning, new algorithms (ensembling techniques maybe ?), or completely new approach to the problem than supervised learning ?
2. How is it possible that only retaining 10% of the vocabulary gives the same scores than the whole ? Is the rest really useless ?
3. Is there a way to efficiently deal with the imbalance in my case, knowing that apparently undersampling is rarely recommended?

Last thing, I don't have the test data so I evaluate with cross validation on the training set, with stratified folds.

Thank you in advance for any insights.

It has been 2 weeks now I am working on SemEval task 4 (2016) : Sentiment Analysis on Twitter.

The results I achieve are lower than what I expected for the three class classification problem : predict if a tweet is negative, neutral or positive. I tried a bunch of classification techniques, all based on the bag-of-words or tf-idf vectorization. Moreover, I added some custom features such as the number of exclamation marks and positive/negative emoticons. I also tried to add a sentiment score thanks to SentiWordNet lexicon. Other lexicons are planned to be used.

Now the point: all of these hand-engineered features do not increase the score, or really slightly. Even things like lemmatization, stopwords removal, and feature selection up to 90% (with chi2 score) do not affect significantly the score.

Finally, I have a class imbalance and I clearly suppose it to affect the score as the confusion matrix and scores for each class (ordered by negative, neutral, positive on the image) show it :

Moreover, when I did undersampling for neutral and positive classes in order to have around the same number of negative samples, accuracy increased by 0.05%.

So, my three questions are :

1. Anyone has ideas to improve my model ? Ideas of new features, text cleaning, new algorithms (ensembling techniques maybe ?), or completely new approach to the problem than supervised learning ?
2. How is it possible that only retaining 10% of the vocabulary gives the same scores than the whole ? Is the rest really useless ?
3. Is there a way to efficiently deal with the imbalance in my case, knowing that apparently undersampling is rarely recommended?

Last thing, I don't have the test data so I evaluate with cross validation on the training set, with stratified folds.

Thank you in advance for any insights.