I have implemented text classification in the sentence level by following through this tutorial. I have used tf-idf and NB & SVM as shown in the tutorial. The code is working fine with my dataset.
With k = 5 i.e. 80% training and 20% testing, I have achieved an accuracy of 70%. However, if I remove the k fold, and just randomly take 80% training and 20% testing after shuffling, it gives me a better result of around 77%.
The implementation for without k-fold CV is as follows:
import numpy as np from sklearn.naive_bayes import MultinomialNB from sklearn.metrics import confusion_matrix from sklearn.svm import LinearSVC from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics import classification_report import random short_pos = open("pos.txt","r").read() short_neg = open("neg.txt","r").read() documents =  for r in short_pos.split('\n'): r= r.rstrip() documents.append(r) for r in short_neg.split('\n'): r= r.rstrip() documents.append(r) stop_words = [unicode(x.strip(), 'utf-8') for x in open("stop_words.txt","r").read().split('\n')] total = 500 * 2 totalFloat= 500.00 * 2 half = total/2; labels = np.zeros(total); labels[0:half] = 1; labels[half:total] = 0; totalNB = 0 totalMatNB = np.zeros((2,2)); indexes = np.arange(0, total); random.shuffle(indexes) train_pc = 80; pc= (total * train_pc )/100 train_index = indexes[:pc]; test_index = indexes[pc:]; X_train = [documents[i] for i in train_index] X_test = [documents[i] for i in test_index] y_train, y_test = labels[train_index], labels[test_index] vectorizer = TfidfVectorizer(min_df=2, use_idf= True, stop_words=stop_words) train_corpus_tf_idf = vectorizer.fit_transform(X_train) test_corpus_tf_idf = vectorizer.transform(X_test) model2 = MultinomialNB() model2.fit(train_corpus_tf_idf, y_train) result2 = model2.predict(test_corpus_tf_idf) totalMatNB = totalMatNB + confusion_matrix(y_test, result2) totalNB = totalNB + sum(y_test == result2) print(classification_report(y_test, result2)) print "NB" print totalMatNB, totalNB/(totalFloat- pc)
The result stays the same [ i.e. better result without k-fold ] for different training and testing ratio.
What could be the reason behind such output?
My dataset range from 400 to 1200 (equal numbers of positive and negative).