0
$\begingroup$

I have the following small dataset (n~140):

  • 1-8 samples per patient
  • A small fraction of samples belonging to the negative class (no tumor), ~ 10-20%
  • A larger fraction of samples belonging to the positive class (tumor)

Each sample belonging to the same patient can be either positive or negative, irrespective of the other samples of the same patient. I need to make sure that samples of the same patient are never both in the training and testset at the same time.

I intend to train a MLP on this data, using repeated 5-fold stratified CV to split into train and testset. I think having a stratified split is important due to the small size of the dataset and the class imbalance. Because my dataset is small, I would like to keep as many samples as I can for training and testing. Can you recommend any way to obtain stratified splits in python that takes my groups (patients) into account?

The only thing I could come up with so far is:

for train_index, test_index in sklearn.model_selection.RepeatedStratifiedKFold(n_splits=5,n_repeats=10,random_state=42).split(traintestset[predictors],traintestset[[response]]):
    test_set=traintestset.iloc[test_index]
    training_set=traintestset.iloc[train_index]

    for patient in list(test_set["patient"].values):
        training_set=training_set[training_set["patient"]!=patient]

    # train a MLP, using cross-validation on the training set

This removes any sample from the training set that belongs to a patient that is also represented by a sample in the test set. However, this way:

  • I lose samples for training
  • The samples in the training set may not be properly stratified anymore.

Would be great if there was a better way to do this. Thank you!

$\endgroup$
0
$\begingroup$

You could start by assigning patients to folds and then take care of stratification within the respective training set.

For stratification, you have two (three) possibilities in general:

  1. exclude samples from over-represented strata - this will inevitably lead to fewer training samples (but that could be done for each surrogate model, so each of the excluded samples is sometimes included, sometimes not)
  2. duplicate samples from the under-represented strata - although a duplicate usually turns out not to be as informative as a real new sample.
    Also, take care that your algorithm doesn't delete duplicates.
  3. Many algorithms/implementations allow to give weights to the samples/cases or specify relative frequencies for the classes. If that's possible for yours, this would be preferred over duplicates.
$\endgroup$
0
$\begingroup$

Thank you cbeleites!

I ended up more or less skipping the stratification part, I only assured that every testset has at least two samples belonging to either group. To do this, I modified the answer from here

def RandomGroupKFold_split(groups, n,traintestset,response, seed=None): 
    #modified from https://stackoverflow.com/a/54254277/11808186
    """
    Random analogous of sklearn.model_selection.GroupKFold.split.

    :return: list of (train, test) indices
    """
    groups = pd.Series(groups)
    ix = np.arange(len(groups))
    unique = np.unique(groups)
    np.random.RandomState(seed).shuffle(unique)
    result = []
    for split in np.array_split(unique, n):
        mask = groups.isin(split)
        train, test = ix[~mask], ix[mask]
        result.append((train, test))

    # This is to ensure that the splitting procedure does not produce a test set with one or less samples of a given category:
    seed_ok=True
    for train_index, test_index in result:
        test_set=traintestset.iloc[test_index]
        if len(test_set[test_set[response]==1][response].values)<2 or len(test_set[test_set[response]==0][response].values)<2:
            seed_ok=False
    if not seed_ok:
        result=RandomGroupKFold_split(groups, n,traintestset, response, seed=seed+1000)
        return result
    else:
        return result

I then also included weights in the classifier downstream as cbeleites suggested.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.