# model performs well on training set via cross-validation, but poorly on hold out set

I'm doing a classification task (just like ad ctr prediction), the data is generated day by day, for example, data from 06-01 to 06-15 is collected, and I split the whole data into 2 parts, the 06-01 to 06-12 as training set, and the 06-13 to 06-15 part as test set.

Over the training set, I build a Logistic Regression model, and I also do cross-validation on this training set when I build the model, the cv score is good, then I re-train the model over the whole training set.

Now I do prediction over the test set using the model I built above, however the result is very bad.

What would be the problem here in my case? I'm sure that there is no strong temporal dependency between the test set and the training set. How should I debug this?

• What metric are you seeking to optimise and why? What is the size of the train set in terms of instances and features? How have you chosen the features retained in the model ? Have you tested the performance against a competitive baseline model and an appropriate null model? – g3o2 Jun 20 '17 at 18:08
• @g3o2 I optimize cross entropy, as per instance size (less than 100k) vs feature size, well there are many categorical features which are handled by feature hashing, and end up with very sparse feature for each row of data, and in total the whole feature space is large than instance size. No feature selection is done except l1 regularization (as well as l2), no baseline model before the one I just built, what is an appropriate null model? – avocado Jun 21 '17 at 0:01
• maybe you should test Naive Bayes as a baseline classifier, which is a priori more comfortable with sparse features. Use the baseline to check whether your model's issues stem from the data or from the classifier. The null model corresponds to what your model is supposed to beat in terms of relevance. In your case, you could try a "random label assigner" with a sampling probability corresponding to the class proportion observed in the training data. By the way, which classification thresholds have you tested so far ? – g3o2 Jun 21 '17 at 11:08
• @g3o2 I'm actually uaing the prediction as ranking, so no specific threshold. – avocado Jun 21 '17 at 13:35
• When mentioned "bad", is it compared to train score or other models? There is a common issue in time-cut data that test data may have a different average level than training set causing poorer score. – Sixiang.Hu Apr 11 '18 at 15:48