I am carrying out a machine learning classification project on some healthcare data. I have at my disposal a training set sampled some months ago in a real life context, and a test set sampled more recently, but following the exact same procedure (same annotator, same software, etc).

As the training dataset is not so large, I decided to use a cross-validation procedure, which outputs pretty good results. However, the final performance on test set is very low.

I then have different questions coming in mind:

  1. I assumed that both training and test set come from the same distribution as they were created the same way. But I'd like to statistically assess this. Plotting boxplots from different features, test set seems very differently distributed than train set, but it could be due to the low number of samples in that set. Is there any test to check if 2 sets are sampled from the same distribution?

  2. To avoid the difference between these sets, I thought to shuffle and mix them both before splitting into 2 new training and test sets. But it seems statistically wrong to me. Also, is normalizing each set separately using Z-score a good practice? It is usually said that normalization should be performed using only the training set mean and std.

  3. I am not sure that my methodology is good. I perform feature selection inside the cross-val loop i.e. the features are selected on each of the fold training set. I then use different subsets of those selected features to train the classifier: the best feature first, then the 2 best features, and so on until I used all features. Plotting some metrics (averaged on the CV folds) against the number of features taken gives me the best number of features I should use to maximize performance. However, the curve oscillates a lot: I thought that it would increase until a certain optimal point and then decrease, but it seems that the best performance is achieved thanks to the randomness of the data.

Example of a curve

Thank you for your answers!

  • $\begingroup$ I don't know what "low number of samples in that set" exactly means, but the problem with this is that if it's really low, any test for equation of distributions will have low power and may not detect anything even if something is going on. This is all the more so for high dimensional data. $\endgroup$ Mar 24 at 13:07

Could you precise:

  • What are (even roughly) the number of samples in the train and test sets?
  • How many features (initially) ?
  • What type of algorithm you are using?

To partly answer your question :

  1. Depending on the type of the distribution you could perform paired Student's t-test for normally distributed features or perform paired Kolmogorov-Smirnov tests that are nonparametric. If the distribution are different then you should first determine if there is no problem in the data collection phase, before addressing the remaining points.
  2. Maybe try to keep a validation set from your training set outside of the cross val. And check the cross val results against the results on this validation set.
  3. Yes, your feature selection curve oscillate a lot. I would suggest that you tune your model so that it has lower variance (and higher bias).

Globally, if you have a large number of features and not that much samples and if you use an highly flexible (low bias & high variance) model, your model will be prone to overfit the train data. So, my answers are to be read in light of the actual number of features, samples and flexibility of the model.

  • $\begingroup$ In the train set: 90 samples, 432 features. In the test set: 14 samples, 432 features. I am using a Random Forest tuned beforehand thanks to a GridSearchCV (5 folds) $\endgroup$
    – aulok
    Mar 24 at 14:41
  • $\begingroup$ @aulok Do you think 14 samples is a sufficient test set size to accurately estimate test loss? $\endgroup$ Mar 24 at 14:48
  • $\begingroup$ Unfortunately, it's the data I've been given, I don't really have control over this. But training set is roughly 85% of all data, and test set 15%, so in between what would give a 5 fold and a 10 fold split $\endgroup$
    – aulok
    Mar 24 at 14:56
  • $\begingroup$ IMO 14 samples for evaluation is too low and explain the oscillation in your plot. Maybe you could collect more samples by doing data augmentation on the MRI images example on KDNuggets $\endgroup$
    – JB RNLT
    Mar 25 at 19:59
  1. How long ago was the training set sampled? Whatever your studying may have a temporal component which left unmodelled will certainly lead to your model failing to internally validate.

a) Z standardize using the training data's mean and sd. The standardization procedure is a mini-model within your model. As such, you use the information you learn from the training set.

b) Do not mix your data sets. Clearly, if your model fails over time then there is something your model is not learning. Shuffling the data now will give you a false sense of confidence and will lead to you performing poorly in the future.

  1. I can't say much about your methodology without more detailed information and an example of your training loop. Are you using AUC as your loss? AUC is not a proper scoring rule, and so perhaps optimizing cross entropy might be a better approach.
  • $\begingroup$ The data I use are texture features extracted from MRI images obtained with a single device. I think that there is no temporal component, however it is a thoughtful idea that I'll investigate. I am using sklearn's GridSearchCV to tune my model, which is a Random Forest, and I am indeed using AUC as the metrics to maximize during the search. $\endgroup$
    – aulok
    Mar 24 at 14:29
  • $\begingroup$ @aulok I'll reiterate that AUC is not a proper scoring rule and you would be best to make probabilistic forecasts rather than classifications. The model is questionable (I don't usually see random forests for NLP problems, unless the feature set is small) but I can't comment on it without more information. It may or may not be a good model, I don't know, but I do know that classifying things is not a good approach generally. $\endgroup$ Mar 24 at 14:32
  • $\begingroup$ For now, I'm doing the following: 1. GridSearch to find the right hyperparameters (bootstrap, max_features, criterion) 2. I perform a cross-val (with the same folds as for the grid search): for each fold training set, I select features with a F-test, and I train a Random Forest using the hyperparameters found thanks to the grid search and an increasing number of selected features. $\endgroup$
    – aulok
    Mar 24 at 14:33
  • $\begingroup$ I'm not answering a NLP problem: the data I'm using are real numbers. I'm trying to classify samples into 2 different classes $\endgroup$
    – aulok
    Mar 24 at 14:35
  • $\begingroup$ @aulok Ah, my mistake. I read "texture" as "textual" and thought you were doing NLP $\endgroup$ Mar 24 at 14:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.