4
$\begingroup$

Curious, do people use data augmentation on the validation set? I am aware there is a debate for the test set -- but the validation set is usually a split form the train set, so wouldn't it make sense to use data augmentation for that?

Also, augmentation helps for models to be better, so wouldn't it make more sense to have the val set version that might improve the model most? (e.g. if you are doing early stopping)


Note: this is not the same as asking to augment the test set. The test should never be used during the ML cycles, only to report values on a paper.


Related:

$\endgroup$
1

4 Answers 4

2
$\begingroup$

I think there is no "one-size-fits-all" answer here.

I use repeated CV always so the question is almost a non-issue in that scenario in my opinion. Not having a single validation fold, if anything allows us to gauge how variable our learner's performance is. So, yes, absolutely I would use data augmentation on the validation set as I see the validation and training set as a natural extension of one another in the case of repeated CV.

The above being said if we notice that we have issues of overfitting one of the things we should investigate is the data augmentation process, on the validation set and as a whole, as it could be the case that we are learning how to generalise against the dataset's augmentation procedure rather than the dataset's generative procedure. Similarly, if we have a single/static validation set, it makes sense not to use augmented data points in it as we will have no way to know if we are biasing our training procedure's validation. Finally do note that by data augmentation we do not have a procedure that alters the specific prevalence of class in our validation set. We should maintain similar class proportions otherwise we will obviously have skewed out-of-sample performance.

$\endgroup$
4
  • 1
    $\begingroup$ What is the basis for data augmentation? $\endgroup$ Commented Mar 7, 2022 at 11:38
  • 1
    $\begingroup$ Usually: insufficient data, privacy concerns, cost concerns, stress testing. Synthetic data generation (a slightly more encompassing term than data augmentation) is gaining momentum exactly because of these. Modelling-wise it can be argued that they help with tackling overfitting/assist generalisation. (Think of how helpful they have been in the context of image recognition.) $\endgroup$
    – usεr11852
    Commented Mar 7, 2022 at 12:55
  • $\begingroup$ How does data augmentation help with privacy concerns? $\endgroup$
    – A. Bollans
    Commented Jun 14, 2022 at 11:26
  • $\begingroup$ Done successfully, it allows us to use a synthetic/artificial population and not share/expose real data directly. $\endgroup$
    – usεr11852
    Commented Jun 14, 2022 at 12:46
2
$\begingroup$

I'd say no. When validating the model you want to estimate its potential performance in the real world. Validating it on synthetic data would work only if the synthetic data is exactly like the data you would find in the wild, but do you ever have such guarantees?

There are different kinds of data augmentation. Things like adding extra noise to images, or rotating them, do not sound controversial from the above point of view, but in such a case you would have the same image repeated in the validation set multiple times, or even worse you would have a different version of the image spread between train and validation sets (data leak). This would bias your results unless you alter the image and throw away the raw image. In most cases, I'd prefer to validate on raw data than on artificially altered data if I needed to choose.

So it is a risky strategy, that can give you an unrealistic estimate of the model performance. Though agree with @usεr11852, there may be cases where you have no better alternatives and poor validation data is better than none.

$\endgroup$
1
  • $\begingroup$ my main issue with validation data augmentation is that usually (at least in DL) you aren't fitting it e.g. no train. So your only choosing a single HP or a set of them with it. So it will increase variance in your estimated value of the val error but since it's not actually being used to fit it's hard to use it to give a useful signal. Thus, one can get the same effect with a normal val set. I think it might help a little but given it's much nosier + its not used to fit -- it's benefits are likely very limitted. $\endgroup$ Commented Jul 15, 2022 at 15:38
1
$\begingroup$

There is a case to be made for using data augmentation on the validation set, as using several copies of the same validation example with different augmentations and averaging their outputs effectively is a type of ensemble method, and ensemble methods in general tend to lead to better performance. This practice is known as test-time augmentation, and was among others used in the original Alexnet paper:

At test time, the network makes a prediction by extracting five 224 × 224 patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions made by the network’s softmax layer on the ten patches.

A clue to why this works is given in the paper Making Convolutional Networks Shift-Invariant Again, in which they show that the output likelihood for the correct class can vary wildly as a function of a shift of the image. When the specific shift gives rise to a low output likelihood, the loss can become very high (especially since it's not linear to the likelihood but logarithmic to it), and it becomes significantly more probable that an incorrect class is predicted. For that reason, it would probably make sense to take several shifts of the image, feed all of them to the network, and take the average of the outputs for the different shifts (even though what they're doing in the paper is probably smarter), which would be a kind of test-time augmentation.

The technique has also been covered by Jason Brownlee here and here.

$\endgroup$
0
$\begingroup$

You can use augmentation data in training, validation and test sets.

The only thing to avoid is using the same data from the training set in validation or test sets.

For example, if you generate 3 augmented instances from an register took from the training data, make sure that no one of these 3 augmented instances accidentally ends up in the validation or test sets.

It turns out that using data from the training set, even augmented data, to validate or test a model is a methodology mistake providing a false generalisation model performance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.