Why does my sentiment analyser only output 1 label? I am trying to do sentiment analysis on a corpus of product reviews. My corpus contains 50,000 samples, of which I take 70% for training and 30% for testing. I discretized the 5-star rating to 3 categories as follows:


*

*[0, 2] = negative

*[3] = neutral

*[4, 5] = positive


As a consequence of this discretization, I see that 42,702 review are positive, 3,230 are negative and 4,068 are neutral.
As features for the classifier, I am using unigrams together with the top 1000 bigram collocations, build using nltk. As classifier, I am using the MultinomialNB from sklearn.
If I evaluate this classifier on my test set, I get the following metrics:
accuracy: 0.8524331858675231
pos precision: 0.8534660677431104
pos recall: 0.9985795454545454
neg precision: 0.19230769230769232
neg recall: 0.0046641791044776115
ntl precision: 1.0
ntl recall: 0.0007412898443291327

As you can see, precision and recall are high, but only for the pos class. If I look at the actual predictions, the classifier always predicts pos. 
Is this because of the uneven distribution of positive/negative/neutral review? How could I further improve upon this?
 A: As the classes in a binary classification problem become more imbalanced, the problem of predicting the most likely class becomes easier. When 85% of the sample is one class, you can achieve 85% accuracy by guessing the modal class every time. So, to get much of an improvement in accuracy over such a trivial model, you need really informative features. If you don't happen to have such features, you're out of luck. Or rather, you're very lucky, since you can get high accuracy with a trivial model that requires no features at all.
Often, models in an unbalanced classification problem are evaluated with a criterion other than simple accuracy, such as a weighted cost function. This often makes sense, but be sure to realize that this isn't a solution to the problem just described. Rather, it's a different problem, possibly a less trivial one where statistical models will be more useful.
A: This is typical when you're predicting something with unbalanced classes which is weak and hard to predict. Essentially, the "cost of being wrong" is causing the model to just choose pos every time, because it is correct 85% of the time and cannot do any better. 


*

*Customize your cost matrix


The best solution is to change the cost matrix so that the penalty for guessing wrong is weighted less for the pos case and more for the others. I don't know how to do that in python, but that is one of the things you should research. 


*Use classifier that specializes with ordered outcomes


Another option is to try ordered or cumulative logit models. Since there is an inherent order in the dependent variable, this violates the assumptions in most of those multinomial sklearn models, and predicting low vs high is equally wrong as predicting low vs medium, and that poor assumption in the cost calculation is helping the model settle on 85%.


*Under Sample pos cases


If you only choose 5 for positive instead of 4,5... You'll have less observations and also be training on the most extreme cases. This might yield better results, but remember that to properly evaluate the model you have to apply the model to the full training set (with the 4s put back in) before judging it's performance. 
