I bulit a naive bayes classifier from 60k vectors of text and each of the text is a 2000 dimension vector(Used bag of words for vectorization). Used simple cross validator to find the best alpha and tested it on the most recent part of the dataset and the accuracy was 78%(dataset has unix timestamp for each record). As per the scikit learn documentation, "A high area under the curve represents both high recall and high precision, where high precision relates to a low false positive rate, and high recall relates to a low false negative rate".

My question is my precision and recall test results are 87% each and the f1 score is 86. Upon constructing a roc curve, Im getting 0.54 as roc. This goes against the words quoted in the scikit documentation. Can someone explain the reason behing it? Is there a way I can improve the roc score and thereby improving the model

con = sqlite3.connect('final.sqlite')

df = pd.read_sql_query('''
''', con)

df = df.sort_values('Time', axis=0)
df.Score.replace(to_replace=[1,2],value=0, inplace=True)
df.Score.replace(to_replace=[4,5], value=1, inplace=True)
CleanedText = df.CleanedText.values[0:100000]

x_train = CleanedText[0:60000]
x_cv = CleanedText[60000:80000]
x_test = CleanedText[80000:100000]

y_train = df.Score.values[0:60000]
y_cv = df.Score.values[60000:80000]
y_test = df.Score.values[80000:100000]

countvect = CountVectorizer(max_features=2000, min_df=15)
train_bowcount = countvect.fit_transform(x_train)
test_bowcount = countvect.fit_transform(x_test)
cv_bowcount = countvect.fit_transform(x_cv)
alpha = [0.000001, 0.00001, 0.0001, 0.001, 0.01, 0.1,1]

for i in alpha:
    nbc = MultinomialNB(alpha=i)
    nbc.fit(train_bowcount, y_train)
    pred = nbc.predict(cv_bowcount)
    accuracy = accuracy_score(y_cv, pred, normalize=True)*float(100)
    print('accuracy for k={} is {}'.format(i, accuracy))

nbc = MultinomialNB(alpha=0.001)
nbc.fit(train_bowcount, y_train)
pred = nbc.predict(test_bowcount)
accuracy = accuracy_score(y_test, pred, normalize=True)*float(100)
print('accuracy for alpha=0.01 in test data is %d' %(accuracy))

EDIT: https://drive.google.com/file/d/18yHOyLnrSgzAabvXoev4C2yjzSThegLB/view?usp=sharing - Dataset

  • $\begingroup$ You need to provide your dataset. Without the data this problem is not reproducible. $\endgroup$ – wind Dec 14 '18 at 6:41
  • $\begingroup$ Sure just a sec!! I have added the link with the question.. please find the same $\endgroup$ – karthikeyan Dec 14 '18 at 6:53
  • $\begingroup$ OK, but you also need to extend the snippet of code with a part loading the database. $\endgroup$ – wind Dec 14 '18 at 7:07
  • $\begingroup$ Yes, I have added it $\endgroup$ – karthikeyan Dec 14 '18 at 7:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.