1
$\begingroup$

I am trying to train a model on text classification. I have a large labeled dataset. Documents are set of comments, notes on a incident. Labels are high level categories for the incidents. As expected, the comments and notes are subjected to human errors, misspellings.

What should be the features for this classification? I have tried TfIdfVectorizer, with tokenizer which uses PorterStemmer. Also i am including ngrams of size 1-4 as features. What additional features can be defined for such a data set?

   df = read_csv(filename, sep="|", na_values=[" "]).fillna(" ")
   le = preprocessing.LabelEncoder()
   target = le.fit_transform(df['label'])

   vectorizer = TfidfVectorizer(sublinear_tf=True, 
                           max_df=0.3,
                           min_df=100,
                           lowercase=True,
                           stop_words='english', 
                           max_features=20000,
                           tokenizer=tokenize,
                           ngram_range=(1,4)
                          )

   train = vectorizer.fit_transform(df['data'])
   X_train, X_test, y_train , y_test = cross_validation.train_test_split(train, target, test_size=5000, random_state=0)
   clf = MultinomialNB(alpha=.1)
   clf.fit(X_train, y_train)
   pred = clf.predict(X_test)

My dataset contains about 300k documents, and vectorizer can produce upto 50k features. I have even tried chisquare to reduce the number of features to 5k, but still accuracy does not improve much. The accuracy was 42% when the data set is 10k or so. The accuracy starts going bad as the data size grows.

Classifiers Tried

SGDClassifier Perceptron PassiveAggressiveClassifier BernoulliNB, MultinomialNB KNeighborsClassifier NearestCentroid RandomForestClassifier

$\endgroup$
  • 1
    $\begingroup$ "The accuracy starts going bad as the data size grows." The annotation of your training data might be noisy. $\endgroup$ – Vladislavs Dovgalecs Nov 1 '15 at 7:05
  • $\begingroup$ @xeon: indeed it is a sign of overfitting. $\endgroup$ – Michael M Nov 1 '15 at 8:33
  • $\begingroup$ sorry how to proceed in this kind of problem? $\endgroup$ – Sumit Chawla Nov 2 '15 at 4:31
  • $\begingroup$ whay you don't try bayesian networks, they do text classifications very well. $\endgroup$ – pmargreff Sep 24 '16 at 9:47
1
$\begingroup$

If you have not done so yet I would suggest to test

 CountVectorizer(binary=True,encoding='utf-8',decode_error='replace',strip_accents='unicode'
                  ,analyzer='word')

and play with the parameters with a grid search

 parameters={'alpha': [1e-2,1e-3,1e-4,1e-5,1e-6,1e-7],'n_iter':[10,20,30,100,200,300] }
 clf=GridSearchCV(estimator=(SGDClassifier(penalty='l2',random_state=42
                  ,class_weight='balanced'))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.