0
$\begingroup$

This question already has an answer here:

I'm green in ML field and I try to classify user reports to valid/invalid. My dataset contains of

  • Valid - 7355 samples
  • Invalid - 6285 samples

So, I devide data into train and test

X_train, X_test, y_train, y_test = train_test_split(descr, reports['label'], 
                                            test_size=0.27, 
                                            random_state=49)

Add some typical stopwords to CountVectorizer and limit max_features to 400 (without this limitation X_train_tfidf.shape was (9957, 181025) and contains a lot of strange features).

count_vect = CountVectorizer(ngram_range=(1, 2), 
                             stop_words =["at", "a", "the"],
                             token_pattern=r"\b\w+\b",  max_features=400)
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer(norm = 'l2',
                                     sublinear_tf=True)
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)


X_test_counts = count_vect.transform(np.asarray(X_test))
X_test_tfidf = tfidf_transformer.transform(X_test_counts)

I tried different classifiers, but the best one was MultinomialNB:

from sklearn.metrics import accuracy_score
from sklearn.naive_bayes import MultinomialNB

nb = MultinomialNB(alpha = 0.7)
nb = nb.fit(X_train_tfidf, y_train)
pred_nb = nb.predict(X_test_tfidf)
accuracy_nb = accuracy_score(y_test, pred_nb)
print('Accuracy for Multinomial Naive Bayes Classifier: ', accuracy_nb)
scores = cross_val_score(nb, X_test_tfidf, y_test, cv=5)
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
print (scores)

Accuracy for Multinomial Naive Bayes Classifier: 0.7656801520499593 Accuracy: 0.76 (+/- 0.03) [0.76693767 0.7394844 0.77038043 0.75951087 0.78125 ]

    precision    recall  f1-score   support

       0       0.76      0.83      0.79      2002
       1       0.77      0.69      0.73      1681

I think a very low rate, spam/ham classification models shows 0.98% accuracy. My Class Prediction Error Visualization looks like this: enter image description here What techniques can I use to improve my results? How can I evaluate my results and understand if they are good enough?

$\endgroup$

marked as duplicate by Michael Chernick, Stephan Kolassa, Peter Flom Feb 11 at 11:26

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.