I am doing binary text classification and I have some feature sets (unigrams, bigrams, dependencies, etc.) and each one of these performs very good individually. For example unigrams alone achieve 89% accuracy and bigrams alone 87% accuracy with 10 Fold CV. But the problem is that when I combine them the accuracy is around 88% which is worse than unigrams alone.

My idea was that by combining good features I will get better results.

Here is one simple Pipeline combining unigrams and bigrams:

    ('features', FeatureUnion(
            ('bot-1', Pipeline([
                ('ext', TextFeatureExtractor(feat_type="bot", term="word", negation=True)),
                ('vec', CountVectorizer(min_df=0.003, max_df=0.9, lowercase=False)),
                ("fs", SelectKBest(score_func=mutual_info_classif, k=10000)),
                ('bin', Binarizer()),
                ('trans', TfidfTransformer(sublinear_tf=True, smooth_idf=True, use_idf=True)),
            ('bot-2', Pipeline([
                ('ext', TextFeatureExtractor(feat_type="bot", term="lemma", negation=True)),
                ('vec', CountVectorizer(min_df=0.002, max_df=0.9, lowercase=False, ngram_range=(2, 2))),
                ("fs", SelectKBest(score_func=mutual_info_classif, k=15000)),
                ('bin', Binarizer()),
                ('trans', TfidfTransformer(sublinear_tf=True, smooth_idf=True, use_idf=True)),
    ('classifier', svm.LinearSVC(C=.5)),

As you can see I do MI Feature Selection and after that I apply tf-idf transformation on the selected features. Am I doing something wrong?

The things I have tried so far:

  • Scale them using StandardScaler(with_mean=False) before classification
  • Scale them using MaxAbsScaler() before classification
  • apply TruncatedSVD with n_components= 100-300 after the TfidfTransformer and also tried normalization after that.

Am I doing something wrong?

  • $\begingroup$ How many observations are in your dataset? Have you tried other models? Have you tried changing the regularization parameter of your SVC? $\endgroup$ – Jason Sanchez Dec 27 '17 at 19:31
  • $\begingroup$ Try normalizing (L_2 normalization) everything, just before applying the classifier. Also, try various values of C for Logistic Regression, e.g., [0.001, 0.01, 0.1, ..,100 ] $\endgroup$ – geompalik Dec 29 '17 at 9:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.