I am trying to train a model on text classification. I have a large labeled dataset. Documents are set of comments, notes on a incident. Labels are high level categories for the incidents. As expected, the comments and notes are subjected to human errors, misspellings.
What should be the features for this classification? I have tried TfIdfVectorizer, with tokenizer which uses PorterStemmer. Also i am including ngrams of size 1-4 as features. What additional features can be defined for such a data set?
df = read_csv(filename, sep="|", na_values=[" "]).fillna(" ") le = preprocessing.LabelEncoder() target = le.fit_transform(df['label']) vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.3, min_df=100, lowercase=True, stop_words='english', max_features=20000, tokenizer=tokenize, ngram_range=(1,4) ) train = vectorizer.fit_transform(df['data']) X_train, X_test, y_train , y_test = cross_validation.train_test_split(train, target, test_size=5000, random_state=0) clf = MultinomialNB(alpha=.1) clf.fit(X_train, y_train) pred = clf.predict(X_test)
My dataset contains about 300k documents, and vectorizer can produce upto 50k features. I have even tried chisquare to reduce the number of features to 5k, but still accuracy does not improve much. The accuracy was 42% when the data set is 10k or so. The accuracy starts going bad as the data size grows.
SGDClassifier Perceptron PassiveAggressiveClassifier BernoulliNB, MultinomialNB KNeighborsClassifier NearestCentroid RandomForestClassifier