I'm green in ML field and I try to classify user reports to valid/invalid. My dataset contains of
- Valid - 7355 samples
- Invalid - 6285 samples
So, I devide data into train and test
X_train, X_test, y_train, y_test = train_test_split(descr, reports['label'], test_size=0.27, random_state=49)
Add some typical stopwords to
CountVectorizer and limit
max_features to 400 (without this limitation
X_train_tfidf.shape was (9957, 181025) and contains a lot of strange features).
count_vect = CountVectorizer(ngram_range=(1, 2), stop_words =["at", "a", "the"], token_pattern=r"\b\w+\b", max_features=400) X_train_counts = count_vect.fit_transform(X_train) tfidf_transformer = TfidfTransformer(norm = 'l2', sublinear_tf=True) X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts) X_test_counts = count_vect.transform(np.asarray(X_test)) X_test_tfidf = tfidf_transformer.transform(X_test_counts)
I tried different classifiers, but the best one was
from sklearn.metrics import accuracy_score from sklearn.naive_bayes import MultinomialNB nb = MultinomialNB(alpha = 0.7) nb = nb.fit(X_train_tfidf, y_train) pred_nb = nb.predict(X_test_tfidf) accuracy_nb = accuracy_score(y_test, pred_nb) print('Accuracy for Multinomial Naive Bayes Classifier: ', accuracy_nb) scores = cross_val_score(nb, X_test_tfidf, y_test, cv=5) print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2)) print (scores)
Accuracy for Multinomial Naive Bayes Classifier: 0.7656801520499593 Accuracy: 0.76 (+/- 0.03) [0.76693767 0.7394844 0.77038043 0.75951087 0.78125 ]
precision recall f1-score support 0 0.76 0.83 0.79 2002 1 0.77 0.69 0.73 1681
I think a very low rate, spam/ham classification models shows 0.98% accuracy. My Class Prediction Error Visualization looks like this: What techniques can I use to improve my results? How can I evaluate my results and understand if they are good enough?