Low performance on test set but high on cross-val I am carrying out a machine learning classification project on some healthcare data.
I have at my disposal a training set sampled some months ago in a real life context, and a test set sampled more recently, but following the exact same procedure (same annotator, same software, etc).
As the training dataset is not so large, I decided to use a cross-validation procedure, which outputs pretty good results. However, the final performance on test set is very low.
I then have different questions coming in mind:

*

*I assumed that both training and test set come from the same distribution as they were created the same way. But I'd like to statistically assess this. Plotting boxplots from different features, test set seems very differently distributed than train set, but it could be due to the low number of samples in that set. Is there any test to check if 2 sets are sampled from the same distribution?


*To avoid the difference between these sets, I thought to shuffle and mix them both before splitting into 2 new training and test sets. But it seems statistically wrong to me. Also, is normalizing each set separately using Z-score a good practice? It is usually said that normalization should be performed using only the training set mean and std.


*I am not sure that my methodology is good. I perform feature selection inside the cross-val loop i.e. the features are selected on each of the fold training set. I then use different subsets of those selected features to train the classifier: the best feature first, then the 2 best features, and so on until I used all features.
Plotting some metrics (averaged on the CV folds) against the number of features taken gives me the best number of features I should use to maximize performance. However, the curve oscillates a lot: I thought that it would increase until a certain optimal point and then decrease, but it seems that the best performance is achieved thanks to the randomness of the data.

Thank you for your answers!
 A: Could you precise:

*

*What are (even roughly) the number of samples in the train and test sets?

*How many features (initially) ?

*What type of algorithm you are using?

To partly answer your question :

*

*Depending on the type of the distribution you could perform paired Student's t-test for normally distributed features or perform paired Kolmogorov-Smirnov tests that are nonparametric. If the distribution are different then you should first determine if there is no problem in the data collection phase, before addressing the remaining points.

*Maybe try to keep a validation set from your training set outside of the cross val. And check the cross val results against the results on this validation set.

*Yes, your feature selection curve oscillate a lot. I would suggest that you tune your model so that it has lower variance (and higher bias).

Globally, if you have a large number of features and not that much samples and if you use an highly flexible (low bias & high variance) model, your model will be prone to overfit the train data. So, my answers are to be read in light of the actual number of features, samples and flexibility of the model.
A: *

*How long ago was the training set sampled?  Whatever your studying may have a temporal component which left unmodelled will certainly lead to your model failing to internally validate.


*

a) Z standardize using the training data's mean and sd.  The standardization procedure is a mini-model within your model. As such, you use the information you learn from the training set.
b) Do not mix your data sets.  Clearly, if your model fails over time then there is something your model is not learning.  Shuffling the data now will give you a false sense of confidence and will lead to you performing poorly in the future.


*I can't say much about your methodology without more detailed information and an example of your training loop.  Are you using AUC as your loss?  AUC is not a proper scoring rule, and so perhaps optimizing cross entropy might be a better approach.

