Cross validation on the train set leads to lower accuracy compared to using a separate test set

Question

I have separate training and test tests normally. When I train my logistic regression model and use it to predict the target class in the test set, the AUC value is 0.75. Below is the python code:

X_train = sklearn.preprocessing.StandardScaler().fit_transform(X_train)    
X_test = sklearn.preprocessing.StandardScaler().fit_transform(X_test)  

classifier = linear_model.LogisticRegressionCV(penalty='l2',
                                               class_weight='balanced',
                                               scoring='roc_auc', 
                                               random_state = 42)
classifier.fit(X_train, y_train)    
predicted = classifier.predict(X_test) 

print "AUC:{}".format(sklearn.metrics.roc_auc_score(y_test, predicted))

>>AUC: 0.75

When I try to see how the optimistic model would perform using 10-fold cross validation on only training set. However, in this case the AUC value is0.56`. I guess it was supposed to be a lot higher since I train and test using the same data set in this case.

X_train = sklearn.preprocessing.StandardScaler().fit_transform(X_train)    

classifier = linear_model.LogisticRegressionCV(penalty='l2',
                                               class_weight='balanced',
                                               scoring='roc_auc', 
                                               random_state = 42)
classifier.fit(X_train, y_train)    
predicted = cross_val_predict(classifier, X_train, y_train, cv = 10) 

print "AUC:{}".format(sklearn.metrics.roc_auc_score(y_train, predicted))

>>AUC: 0.56

Please note that the target label is 1 or 0. Also, the training set is let's say 2nd month behavior of users, whereas test data is the 3rd month behavior. So, it is actually not like dividing the same data set into test and training. I expected lower accuracy when predicting future behavior with a model trained on past behavior.

I assume that these results are not possible with no error in my code. Do you see any problems with my code? Or, are these results ever possible? Then, what could be the reason?

UPDATE

Here is a visualization to explain what I have been trying to do along with the class distribution. So, I have 3 time points: TP 0, TP 1, and TP 2. Using the information regarding user activities only between TP 0 - TP 1, I train a model to predict a specific user behavior (let's say y), then I use this model to predict the same user behavior that took place at TP 2.

I have also repeated the same experiment when training and test sets are swapped. In this case the CV on the training set seems to produce higher accuracy.One thing we assume is that overall user activities and user behavior that we predict gets more stable over time. Maybe, I am having these results because of this nature of the data at hand.

Answer 1

The size (both no. of cases in train and test sets and no. of features) would be needed to judge how likely different possible explanations are.

• First of all, while I'm not familiar with scikit learn your code looks to me as if you do the feature scaling separately on training and test set, i.e. the test set scaling factors are calculated from the test set. That would be wrong: the test set features should be centered and scaled with the offset and factors estimated from the training set.
  I find it helpful to think of these "preprocessing" steps as part of the model. Also, you could have a test set consisting of only 1 case. How would you then scale?

  • If both training and test set are large enough, the same scaling parameters are calculated and the error is not apparent.

  This also means that the scaling should be calculated for each of the cross validation surrogate models. Which doesn't seem to be the case here (correct me if I'm wrong) and would lead to an optimistic bias in the cross validation results.

  • If your training set is large enough, this optimistic bias in the CV results becomes negligible

• If you have relatively few cases, the performance estimate (AUC as well as others) suffers from variance uncertainty. It is good practice to check out this variance and take it into accout for comparison.

• Properly set up cross validation which is not used for training (tuning, optimzation, hyperparameter estimation) is expected to have a small pessimistic bias. In case of leave one out CV, the bias can be large and pessimistic as always a class is tested that is underrepresented in the respective training set.

• And yes, it may happen that your test set is easier than the training set. You can easily test that idea here: just swap training and test sets.
  This would lead to the conclusion that your data is not representative.