# Python Text Classification Features Engineering

I am trying to train a model on text classification. I have a large labeled dataset. Documents are set of comments, notes on a incident. Labels are high level categories for the incidents. As expected, the comments and notes are subjected to human errors, misspellings.

What should be the features for this classification? I have tried TfIdfVectorizer, with tokenizer which uses PorterStemmer. Also i am including ngrams of size 1-4 as features. What additional features can be defined for such a data set?

   df = read_csv(filename, sep="|", na_values=[" "]).fillna(" ")
le = preprocessing.LabelEncoder()
target = le.fit_transform(df['label'])

vectorizer = TfidfVectorizer(sublinear_tf=True,
max_df=0.3,
min_df=100,
lowercase=True,
stop_words='english',
max_features=20000,
tokenizer=tokenize,
ngram_range=(1,4)
)

train = vectorizer.fit_transform(df['data'])
X_train, X_test, y_train , y_test = cross_validation.train_test_split(train, target, test_size=5000, random_state=0)
clf = MultinomialNB(alpha=.1)
clf.fit(X_train, y_train)
pred = clf.predict(X_test)


My dataset contains about 300k documents, and vectorizer can produce upto 50k features. I have even tried chisquare to reduce the number of features to 5k, but still accuracy does not improve much. The accuracy was 42% when the data set is 10k or so. The accuracy starts going bad as the data size grows.

Classifiers Tried

SGDClassifier Perceptron PassiveAggressiveClassifier BernoulliNB, MultinomialNB KNeighborsClassifier NearestCentroid RandomForestClassifier

• "The accuracy starts going bad as the data size grows." The annotation of your training data might be noisy. – Vladislavs Dovgalecs Nov 1 '15 at 7:05
• @xeon: indeed it is a sign of overfitting. – Michael M Nov 1 '15 at 8:33
• sorry how to proceed in this kind of problem? – Sumit Chawla Nov 2 '15 at 4:31
• whay you don't try bayesian networks, they do text classifications very well. – pmargreff Sep 24 '16 at 9:47

 CountVectorizer(binary=True,encoding='utf-8',decode_error='replace',strip_accents='unicode'

 parameters={'alpha': [1e-2,1e-3,1e-4,1e-5,1e-6,1e-7],'n_iter':[10,20,30,100,200,300] }