I have the following small dataset (n~140):
- 1-8 samples per patient
- A small fraction of samples belonging to the negative class (no tumor), ~ 10-20%
- A larger fraction of samples belonging to the positive class (tumor)
Each sample belonging to the same patient can be either positive or negative, irrespective of the other samples of the same patient. I need to make sure that samples of the same patient are never both in the training and testset at the same time.
I intend to train a MLP on this data, using repeated 5-fold stratified CV to split into train and testset. I think having a stratified split is important due to the small size of the dataset and the class imbalance. Because my dataset is small, I would like to keep as many samples as I can for training and testing. Can you recommend any way to obtain stratified splits in python that takes my groups (patients) into account?
The only thing I could come up with so far is:
for train_index, test_index in sklearn.model_selection.RepeatedStratifiedKFold(n_splits=5,n_repeats=10,random_state=42).split(traintestset[predictors],traintestset[[response]]): test_set=traintestset.iloc[test_index] training_set=traintestset.iloc[train_index] for patient in list(test_set["patient"].values): training_set=training_set[training_set["patient"]!=patient] # train a MLP, using cross-validation on the training set
This removes any sample from the training set that belongs to a patient that is also represented by a sample in the test set. However, this way:
- I lose samples for training
- The samples in the training set may not be properly stratified anymore.
Would be great if there was a better way to do this. Thank you!