I am carrying out a machine learning classification project on some healthcare data. I have at my disposal a training set sampled some months ago in a real life context, and a test set sampled more recently, but following the exact same procedure (same annotator, same software, etc).
As the training dataset is not so large, I decided to use a cross-validation procedure, which outputs pretty good results. However, the final performance on test set is very low.
I then have different questions coming in mind:
I assumed that both training and test set come from the same distribution as they were created the same way. But I'd like to statistically assess this. Plotting boxplots from different features, test set seems very differently distributed than train set, but it could be due to the low number of samples in that set. Is there any test to check if 2 sets are sampled from the same distribution?
To avoid the difference between these sets, I thought to shuffle and mix them both before splitting into 2 new training and test sets. But it seems statistically wrong to me. Also, is normalizing each set separately using Z-score a good practice? It is usually said that normalization should be performed using only the training set mean and std.
I am not sure that my methodology is good. I perform feature selection inside the cross-val loop i.e. the features are selected on each of the fold training set. I then use different subsets of those selected features to train the classifier: the best feature first, then the 2 best features, and so on until I used all features. Plotting some metrics (averaged on the CV folds) against the number of features taken gives me the best number of features I should use to maximize performance. However, the curve oscillates a lot: I thought that it would increase until a certain optimal point and then decrease, but it seems that the best performance is achieved thanks to the randomness of the data.
Thank you for your answers!