I have separate training and test tests normally. When I train my logistic regression model and use it to predict the target class in the test set, the AUC
value is 0.75
. Below is the python code:
X_train = sklearn.preprocessing.StandardScaler().fit_transform(X_train)
X_test = sklearn.preprocessing.StandardScaler().fit_transform(X_test)
classifier = linear_model.LogisticRegressionCV(penalty='l2',
class_weight='balanced',
scoring='roc_auc',
random_state = 42)
classifier.fit(X_train, y_train)
predicted = classifier.predict(X_test)
print "AUC:{}".format(sklearn.metrics.roc_auc_score(y_test, predicted))
>>AUC: 0.75
When I try to see how the optimistic model would perform using 10-fold cross validation on only training set. However, in this case the AUC value is
0.56`. I guess it was supposed to be a lot higher since I train and test using the same data set in this case.
X_train = sklearn.preprocessing.StandardScaler().fit_transform(X_train)
classifier = linear_model.LogisticRegressionCV(penalty='l2',
class_weight='balanced',
scoring='roc_auc',
random_state = 42)
classifier.fit(X_train, y_train)
predicted = cross_val_predict(classifier, X_train, y_train, cv = 10)
print "AUC:{}".format(sklearn.metrics.roc_auc_score(y_train, predicted))
>>AUC: 0.56
Please note that the target label is 1 or 0. Also, the training set is let's say 2nd month behavior of users, whereas test data is the 3rd month behavior. So, it is actually not like dividing the same data set into test and training. I expected lower accuracy when predicting future behavior with a model trained on past behavior.
I assume that these results are not possible with no error in my code. Do you see any problems with my code? Or, are these results ever possible? Then, what could be the reason?
UPDATE
Here is a visualization to explain what I have been trying to do along with the class distribution. So, I have 3 time points: TP 0, TP 1, and TP 2. Using the information regarding user activities only between TP 0 - TP 1, I train a model to predict a specific user behavior (let's say y
), then I use this model to predict the same user behavior that took place at TP 2.
I have also repeated the same experiment when training and test sets are swapped. In this case the CV on the training set seems to produce higher accuracy.One thing we assume is that overall user activities and user behavior that we predict gets more stable over time. Maybe, I am having these results because of this nature of the data at hand.
classifier.fit(X, y_train)
in the first example andclassifier.fit(X_train, y_train)
in the second? $\endgroup$