# Using k fold cross validation gives lower results than without using it

I have implemented text classification in the sentence level by following through this tutorial. I have used tf-idf and NB & SVM as shown in the tutorial. The code is working fine with my dataset.

With k = 5 i.e. 80% training and 20% testing, I have achieved an accuracy of 70%. However, if I remove the k fold, and just randomly take 80% training and 20% testing after shuffling, it gives me a better result of around 77%.

The implementation for without k-fold CV is as follows:

import numpy as np
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import confusion_matrix
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import classification_report
import random

documents = []

for r in short_pos.split('\n'):
r= r.rstrip()

documents.append(r)

for r in short_neg.split('\n'):
r= r.rstrip()

documents.append(r)

stop_words = [unicode(x.strip(), 'utf-8') for x in open("stop_words.txt","r").read().split('\n')]

total = 500 * 2
totalFloat= 500.00 * 2
half = total/2;

labels = np.zeros(total);
labels[0:half] = 1;
labels[half:total] = 0;

totalNB = 0
totalMatNB = np.zeros((2,2));

indexes = np.arange(0, total);
random.shuffle(indexes)

train_pc = 80;
pc= (total * train_pc )/100
train_index = indexes[:pc];
test_index = indexes[pc:];

X_train = [documents[i] for i in train_index]
X_test = [documents[i] for i in test_index]

y_train, y_test = labels[train_index], labels[test_index]

vectorizer = TfidfVectorizer(min_df=2, use_idf= True, stop_words=stop_words)
train_corpus_tf_idf = vectorizer.fit_transform(X_train)
test_corpus_tf_idf = vectorizer.transform(X_test)
model2 = MultinomialNB()
model2.fit(train_corpus_tf_idf, y_train)
result2 = model2.predict(test_corpus_tf_idf)
totalMatNB = totalMatNB + confusion_matrix(y_test, result2)
totalNB = totalNB + sum(y_test == result2)

print(classification_report(y_test, result2))

print "NB"
print totalMatNB, totalNB/(totalFloat- pc)


The result stays the same [ i.e. better result without k-fold ] for different training and testing ratio.

What could be the reason behind such output?

My dataset range from 400 to 1200 (equal numbers of positive and negative).

• accuracy on what? On an additional independent data set? – Michael M May 11 '18 at 14:24
• @MichaelM I am new in this field, I am just following the code in the tutorial. First i run it by just changing by the data set and its size. So, its the accuracy on the testing dataset. Next i tried the same by simply removing the k-fold CV, and run it for only one iteration. The later gave better result – OnePunchMan May 11 '18 at 14:41
• Empirical information is finite, i.e., random samples drawn from the same data using different methods will yield differing, approximately equivalent results. In other words I would argue that the findings from the two approaches are similar if not the same and lead to the same strategic decisions. – Mike Hunter May 11 '18 at 15:18
• @DJohnson please look at the edit, i have added the implementation of without using k-fold CV – OnePunchMan May 11 '18 at 15:19

It could be just chance, meaning that that particular 80-20 split can be "easier to learn" than the average. How do you get the average? By increasing your cross-validation $k$ to a value such that your accuracy remains stable, in the sense that if you repeat the procedure you get a similar accuracy. You can try to do a second iteration of your 80-20 split, and check the accuracy, very probably it will be lower than 77%, and probably lower than 70%. Also, remember that if you are using a random seed your first split will always be the same, even if you run it multiple times.