Python Text Classification Features Engineering

Question

I am trying to train a model on text classification. I have a large labeled dataset. Documents are set of comments, notes on a incident. Labels are high level categories for the incidents. As expected, the comments and notes are subjected to human errors, misspellings.

What should be the features for this classification? I have tried TfIdfVectorizer, with tokenizer which uses PorterStemmer. Also i am including ngrams of size 1-4 as features. What additional features can be defined for such a data set?

   df = read_csv(filename, sep="|", na_values=[" "]).fillna(" ")
   le = preprocessing.LabelEncoder()
   target = le.fit_transform(df['label'])

   vectorizer = TfidfVectorizer(sublinear_tf=True, 
                           max_df=0.3,
                           min_df=100,
                           lowercase=True,
                           stop_words='english', 
                           max_features=20000,
                           tokenizer=tokenize,
                           ngram_range=(1,4)
                          )

   train = vectorizer.fit_transform(df['data'])
   X_train, X_test, y_train , y_test = cross_validation.train_test_split(train, target, test_size=5000, random_state=0)
   clf = MultinomialNB(alpha=.1)
   clf.fit(X_train, y_train)
   pred = clf.predict(X_test)

My dataset contains about 300k documents, and vectorizer can produce upto 50k features. I have even tried chisquare to reduce the number of features to 5k, but still accuracy does not improve much. The accuracy was 42% when the data set is 10k or so. The accuracy starts going bad as the data size grows.

Classifiers Tried

SGDClassifier Perceptron PassiveAggressiveClassifier BernoulliNB, MultinomialNB KNeighborsClassifier NearestCentroid RandomForestClassifier

Answer 1

If you have not done so yet I would suggest to test

 CountVectorizer(binary=True,encoding='utf-8',decode_error='replace',strip_accents='unicode'
                  ,analyzer='word')

and play with the parameters with a grid search

 parameters={'alpha': [1e-2,1e-3,1e-4,1e-5,1e-6,1e-7],'n_iter':[10,20,30,100,200,300] }
 clf=GridSearchCV(estimator=(SGDClassifier(penalty='l2',random_state=42
                  ,class_weight='balanced'))