I have implemented text classification in the sentence level by following through this tutorial. I have used tf-idf and NB & SVM as shown in the tutorial. The code is working fine with my dataset.

With k = 5 i.e. 80% training and 20% testing, I have achieved an accuracy of 70%. However, if I remove the k fold, and just randomly take 80% training and 20% testing after shuffling, it gives me a better result of around 77%.

The implementation for without k-fold CV is as follows:

import numpy as np
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import confusion_matrix
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import classification_report
import random

short_pos = open("pos.txt","r").read()
short_neg = open("neg.txt","r").read()  

documents = []

for r in short_pos.split('\n'):
    r= r.rstrip()


for r in short_neg.split('\n'):
    r= r.rstrip()


stop_words = [unicode(x.strip(), 'utf-8') for x in open("stop_words.txt","r").read().split('\n')]

total = 500 * 2
totalFloat= 500.00 * 2
half = total/2;

labels = np.zeros(total);
labels[0:half] = 1;
labels[half:total] = 0;

totalNB = 0
totalMatNB = np.zeros((2,2));

indexes = np.arange(0, total);

train_pc = 80;
pc= (total * train_pc )/100
train_index = indexes[:pc];
test_index = indexes[pc:];

X_train = [documents[i] for i in train_index]
X_test = [documents[i] for i in test_index]

y_train, y_test = labels[train_index], labels[test_index]

vectorizer = TfidfVectorizer(min_df=2, use_idf= True, stop_words=stop_words)
train_corpus_tf_idf = vectorizer.fit_transform(X_train)
test_corpus_tf_idf = vectorizer.transform(X_test)
model2 = MultinomialNB()
model2.fit(train_corpus_tf_idf, y_train)
result2 = model2.predict(test_corpus_tf_idf)
totalMatNB = totalMatNB + confusion_matrix(y_test, result2)
totalNB = totalNB + sum(y_test == result2)

print(classification_report(y_test, result2))

print "NB"
print totalMatNB, totalNB/(totalFloat- pc)

The result stays the same [ i.e. better result without k-fold ] for different training and testing ratio.

What could be the reason behind such output?

My dataset range from 400 to 1200 (equal numbers of positive and negative).

  • $\begingroup$ accuracy on what? On an additional independent data set? $\endgroup$ – Michael M May 11 '18 at 14:24
  • $\begingroup$ @MichaelM I am new in this field, I am just following the code in the tutorial. First i run it by just changing by the data set and its size. So, its the accuracy on the testing dataset. Next i tried the same by simply removing the k-fold CV, and run it for only one iteration. The later gave better result $\endgroup$ – OnePunchMan May 11 '18 at 14:41
  • $\begingroup$ Empirical information is finite, i.e., random samples drawn from the same data using different methods will yield differing, approximately equivalent results. In other words I would argue that the findings from the two approaches are similar if not the same and lead to the same strategic decisions. $\endgroup$ – Mike Hunter May 11 '18 at 15:18
  • $\begingroup$ @DJohnson please look at the edit, i have added the implementation of without using k-fold CV $\endgroup$ – OnePunchMan May 11 '18 at 15:19

It could be just chance, meaning that that particular 80-20 split can be "easier to learn" than the average. How do you get the average? By increasing your cross-validation $k$ to a value such that your accuracy remains stable, in the sense that if you repeat the procedure you get a similar accuracy. You can try to do a second iteration of your 80-20 split, and check the accuracy, very probably it will be lower than 77%, and probably lower than 70%. Also, remember that if you are using a random seed your first split will always be the same, even if you run it multiple times.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.