0
$\begingroup$

I'm trying to use data augmentation for training a model for a classification task. But I'm not sure about how to use data augmentation in a fair and meaningful way in the evaluation of a machine learning model. I have considered at least three different options for doing this and I would like to know which one is the best.

Firt of all, let us suppose that we have a labeled dataset D.

Option 1:

-We can use a data augmentation approach over D, for generating a novel set of synthetic data S.
-We can create a new dataset D2, which combines D and S.
-With D2, we can use a 10-fold cross-validation approach, where:
    -We divide D2 into 10 random subsets (folds).
    -We use each fold as test data and the remaining 9 folds as training data.
    -This is repeated for each fold.
    -We obtain the average performance (on the 10 folds) for evaluating the model.

Option 2:

-We can use a data augmentation approach over D, for generating a novel set of synthetic data S.
-With D, we can use a 10-fold cross-validation approach, where:
    -We divide D into 10 random subsets (folds).
    -We use each fold as test.
    -We create a set T including the remaining 9 folds.
    -We create a new set T2 that combines T and S
    -We use T2 as the training data.
   -This is repeated for each fold.
   -We obtain the average performance (on the 10 folds) for evaluating the model.

Option 3:

-From D, we can use a 10-fold cross-validation approach, where:
       -We divide D into 10 random subsets (folds).
       -We use each fold as test.
       -We create a set T including the remaining 9 folds.
       -We use a data augmentation approach over T for generating a set S of synthetic samples.
       -We create a set T2 that combines T and S.
       -We use T2 as the training data.
       -This is repeated for each fold.
       -We obtain the average performance for evaluating the model.

What is the best option and why?

$\endgroup$

1 Answer 1

2
$\begingroup$

Option 3.

Options 1 and 2 have an information bleed-through issue between your train/validation data. Because you are creating S BEFORE splitting into folds, you will have synthetic data that was created from observations in your validation fold. By having data that is in some way only a slight permutation of your validation data, you haven't properly separated your validation/train data.

Option 3 is the only one that does not have the bleed-through issue because S is only created based on T.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.