I did a Multi-class document classification. I divided the original data set (18,8334 documents as a list of strings where each element of list is a document string.) into 70% training and 30% test.

Then on the 70% training dataset, I used sklearn 5 fold cross validation to train the model. I used three models. First was Gaussian Naive Bayes, second was Random Forests and third was Stochastic Gradient Descent SVM.

Stochastic gradient descent gave the highest cross validated accuracy of 0.85. But the same model when tested on the 30% test dataset gives 9% accuracy. Why is that? Isn't cross-validation error a measure or estimate of test error/generalization error?



This is how I created the train_test(70/30)

def split(docs_list,target_recoded):
    """This function samples the dataset into training and testing"""
    # Splitting into training and test. 
    from sklearn.cross_validation import train_test_split
    train_X, test_X,train_Y,test_Y = train_test_split(docs_list, target_recoded, test_size=0.30, random_state=42)

    return train_X, test_X,train_Y,test_Y

After initial nlp preprocessing like stop words removal, stemming etc, I have a cleaned list of doc strings. On that, I used the following for bag of words creation. First 70% training data is passed and then 30% test data was passed as argument to this function.

def bagofWords(X,Y,max_feature=5000,type="count"):
    """This function creates a Bag of Features vectors from the original documents"""

    from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
    # Initialize the "CountVectorizer" object, which is scikit-learn's
    # bag of words tool.  

    if(type=="count"): # To choose between count or tf-idf bag or words model
        vectorizer = CountVectorizer(analyzer = "word",max_features = max_feature) 
        vectorizer = TfidfVectorizer(analyzer = "word",max_features = max_feature)

    return X ,np.array(Y) 

This is how I train an SGD

 def SGD(self):
        """Method to implement Multi-class SVM using Stochastic Gradient Descent"""

        from sklearn.linear_model import SGDClassifier
        scores_sgd = []

        for train_indices, test_indices in self.k_fold:
            train_X_cv = self.train_X[train_indices].todense()
            train_Y_cv= self.train_Y[train_indices]

            test_X_cv = self.train_X[test_indices].todense()
            test_Y_cv= self.train_Y[test_indices]


        print("The mean accuracy of Stochastic Gradient Descent Classifier on CV data is:", np.mean(scores_sgd))   

And this is to check test performance

def test_performance(self,test_X,test_Y):
        """This method checks the performance of each algorithm on test data."""

        from sklearn import metrics
# For SGD
        print ("The accuracy of SGD on test data is:", self.sgd.score(test_X,test_Y))
        print 'Classification Metrics for SGD'
        print metrics.classification_report(test_Y, self.sgd.predict(test_X))
        print "Confusion matrix"
        print metrics.confusion_matrix(test_Y, self.sgd.predict(test_X))
  • 1
    $\begingroup$ Is 9% accuracy a typo? If not, could you have switched your test data labels around at all? $\endgroup$ – jld May 11 '15 at 17:20
  • 2
    $\begingroup$ Yeah, it sounds like probably a code mistake. Are you preprocessing the test data in a consistent way to the training? (In scikit-learn, keeping the same transformers and calling transform but not fit.) $\endgroup$ – Danica May 11 '15 at 17:26
  • $\begingroup$ Here is the thing. I have done the similar data preprocessing. But for converting the test documents to bag of words, I have used again fit_transform. Is this ok? Or should I use the earlier fit_transform in vectorizer for train data $\endgroup$ – Baktaawar May 11 '15 at 18:45
  • $\begingroup$ Pls check edit. I have added the steps I did $\endgroup$ – Baktaawar May 11 '15 at 18:53

Hastie et al discuss this precise issue in their book, The Elements of Statistical Learning. They conclude that cross-validation is NOT an estimate of test-error conditional on the training set. Rather, they believe it is an estimate of the unconditional test error. In other words, this is the expected test error if you are also randomizing over the world of possible training sets, rather than the precise training set you've been given.

  • $\begingroup$ Could you please elaborate on this a bit? $\endgroup$ – Baktaawar May 11 '15 at 18:46
  • 4
    $\begingroup$ The book is free online, search for The Elements of Statistical Learning. This issue is mentioned at the very beginning of chapter 7. Also see 7.4 and 7.10. $\endgroup$ – Matthew Drury May 11 '15 at 19:06

I'd suspect something is wrong with how you created the hold out set. Ie either it wasn't randomized or needed to be done with stratified sampling.

This sort of thing can also happen if you did feature selection or engineering outside of the CV loop as you'll produce features that look good across the cv folds but don't generalize.

It can also happen if you tuned model hyperparameters using the CV loop. How did the other model's do on the hold out set? SVMs can be quite prone to this while RF's tend to be less prone so i'd wonder about that in particular. Other strategies for dealing with this include an inner CV loop and fixing the regularization parameters to limit the amount of overfitting you can do.

Finally if this is highly dimensional data with lots of variance unrelated to the target (like genetic data) it is quite possible there exist models that works well in the cv data but not the hold out data by random chance.

This phenomena has been described as "anti learning", again RF's tend to be more robust against this (because of the internal bagging) though it begins to effect them as well as the dimensionality grows.

  • $\begingroup$ Pls check edit. Have added the steps I followed $\endgroup$ – Baktaawar May 11 '15 at 18:53
  • 3
    $\begingroup$ The way you are doing bag of words is suspect. For both the hold out set and each cv fold you need to call fit_transform only on what you are training the model on, and just call transform when you do the testing. You can use a sklearn.pipeline.Pipeline to treat preprocessing as part of your model to enforce this. As is, it is likely producing different vectors/order of vectors on the 30% hold out and possibly overfittiting by fitting on the entire 70% outside of the folds for the cross validation. $\endgroup$ – Ryan Bressler May 11 '15 at 20:28
  • $\begingroup$ @RyanBressler, this is definitely the reason for the huge gap. You should turn your comment into an answer and I'll upvote it. :p $\endgroup$ – Danica May 11 '15 at 21:17
  • $\begingroup$ Ok. So the way I understand this, I need to just do transform on the test set. I think I kind of missed it. I guess it is creating bag of words on the reduced test dataset by again learning it. So obviously it would have different feature vectors. Does this sound correct? $\endgroup$ – Baktaawar May 11 '15 at 21:34
  • 3
    $\begingroup$ Yes. It was correct. Just removed fit_transform to transform. And now the test accuracy is same as cross val. 85% for SGD and 76% for Random Forests. I haven't tweaked a lot of hyper parameters for RF, so it's like with default settings. But yes, the problem was that I mistakably fitted new tfidf vectors using test data, hence obviously lot of feature vectors would be different as the test_data is just 30% of whole data. $\endgroup$ – Baktaawar May 11 '15 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.