I did a Multi-class document classification. I divided the original data set (18,8334 documents as a list of strings where each element of list is a document string.) into 70% training and 30% test.
Then on the 70% training dataset, I used sklearn 5 fold cross validation to train the model. I used three models. First was Gaussian Naive Bayes, second was Random Forests and third was Stochastic Gradient Descent SVM.
Stochastic gradient descent gave the highest cross validated accuracy of 0.85. But the same model when tested on the 30% test dataset gives 9% accuracy. Why is that? Isn't cross-validation error a measure or estimate of test error/generalization error?
This is how I created the train_test(70/30)
def split(docs_list,target_recoded): """This function samples the dataset into training and testing""" # Splitting into training and test. from sklearn.cross_validation import train_test_split train_X, test_X,train_Y,test_Y = train_test_split(docs_list, target_recoded, test_size=0.30, random_state=42) return train_X, test_X,train_Y,test_Y
After initial nlp preprocessing like stop words removal, stemming etc, I have a cleaned list of doc strings. On that, I used the following for bag of words creation. First 70% training data is passed and then 30% test data was passed as argument to this function.
def bagofWords(X,Y,max_feature=5000,type="count"): """This function creates a Bag of Features vectors from the original documents""" from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer # Initialize the "CountVectorizer" object, which is scikit-learn's # bag of words tool. if(type=="count"): # To choose between count or tf-idf bag or words model vectorizer = CountVectorizer(analyzer = "word",max_features = max_feature) else: vectorizer = TfidfVectorizer(analyzer = "word",max_features = max_feature) X=vectorizer.fit_transform(X) return X ,np.array(Y)
This is how I train an SGD
def SGD(self): """Method to implement Multi-class SVM using Stochastic Gradient Descent""" from sklearn.linear_model import SGDClassifier scores_sgd =  for train_indices, test_indices in self.k_fold: train_X_cv = self.train_X[train_indices].todense() train_Y_cv= self.train_Y[train_indices] test_X_cv = self.train_X[test_indices].todense() test_Y_cv= self.train_Y[test_indices] self.sgd=SGDClassifier(loss='hinge',penalty='l2') scores_sgd.append(self.sgd.fit(train_X_cv,train_Y_cv).score(test_X_cv,test_Y_cv)) print("The mean accuracy of Stochastic Gradient Descent Classifier on CV data is:", np.mean(scores_sgd))
And this is to check test performance
def test_performance(self,test_X,test_Y): """This method checks the performance of each algorithm on test data.""" from sklearn import metrics # For SGD print ("The accuracy of SGD on test data is:", self.sgd.score(test_X,test_Y)) print 'Classification Metrics for SGD' print metrics.classification_report(test_Y, self.sgd.predict(test_X)) print "Confusion matrix" print metrics.confusion_matrix(test_Y, self.sgd.predict(test_X))