2
$\begingroup$

Problem

I am trying to do sentiment analysis using pretrained word vectors GloVe, which is essentially a look-up table that maps word to a fix-dimension vector. Since GloVe is initially designed to provide word embedding for each individual word, for the entire document, I take the average of the all words and use the resulting averaged vector to represent the document. Then the vectors are treated as features and sent to SVM for classification.

I know this approach might not provide satisfying result, surprisingly, it performs very poor (see below).

  • In terms of accuracy, it is almost like random guess.
  • The F1 score is extremely low . More surprisingly, true negative rate (TNR) is almost 1 but true positive rate (TPR) is almost 0 (three confusion matrices are similar and only one is included here).

enter image description here enter image description here I kind of understand why the accuracy is low after I did some visualizations of my document vectors using TSNE (see below). But I do not understand why the predictor will be biased to negative samples. f

Updates:

I did make the dataset balanced before any processing. There are 10000 positive samples and 10000 negative samples.

$\endgroup$
  • $\begingroup$ what is the class imbalance in the dataset? it sounds like the negative label dominates. $\endgroup$ – Zhubarb Jun 5 '19 at 7:27
  • $\begingroup$ Seems like your model did not learn anything, predicts all as negative, and since your dataset is probably 50% positive and 50% negative, it gets 50% accuracy (guessing the negatives correctly). $\endgroup$ – Jan Kukacka Jun 5 '19 at 7:35
  • $\begingroup$ @Zhubarb The dataset is made balanced before processing. $\endgroup$ – Mr.Robot Jun 5 '19 at 7:38
  • $\begingroup$ @JanKukacka Yes. But I am not sure why this will happen. Especially, if a model does not learn anything, should not it predict positive/negative samples equally bad, which will result in about 0.5 TPR and 0.5 TNR. $\endgroup$ – Mr.Robot Jun 5 '19 at 7:40
1
$\begingroup$

As you point out an accuracy of 0.5 (on average) is the lowest you could get by random labeling. A very low F1 score and an accuracy of 0.5 are therefore not logically inconsistent. I would recommend rephrasing the question title. Your question is "why does the classifier predict everything as negative?". If the class imbalance does not exist (no error in your pre-processing), that is indeed surprising.

At first glance, your confusion matrix is indicative of a highly imbalanced data set whereby the classifier labels virtually all observations as negative. You mention you manually down or up sample to ensure equal class distribution beforehand. My guess is that there is something wrong with this pre-processing. For instance do you apply this sampling equally to train/validation and test sets? (Mind you, In real-life you do not have knowledge or control of the test set).

Why not try with a simple classifier first and what output you get. There should be no reason a classifier is biased towards a particular label.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.