Data augmentation on training set only?

Question

Is it common practice to apply data augmentation to training set only, or to both training and test sets?

Answer 1

In terms of the concept of augmentation, ie making the data set bigger for some reason, we'd tend to only augment the training set. We'd evaluate the result of different augmentation approaches on a validation set.

However, as @Łukasz Grad points out, we might need to perform a similar procedure to the test set as was done on the training set. This is typically so that the input data from the test set resembles as much as possible that of the training set. For example, @Łukasz Grad points out the example of image cropping, where we'd need to crop the test images too, so they are the same size as the training images. However, in the case of the training images, we might use each training image multiple times, with crops in different locations/offsets. At test time we'd likely either do a single centred crop, or do random crops and take an average.

Running the augmentation procedure against test data is not to make the test data bigger/more accurate, but just to make the input data from the test set resemble that of the input data from the training set, so we can feed it into the same net (eg same dimensions). We'd never consider that the test set is 'better' in some way, by applying an augmentation procedure. At least, that's not something I've ever seen.

On the other hand, for the training set, the point of the augmentation is to reduce overfitting during training. And we evaluate the quality of the augmentation by then running the trained model against our more-or-less fixed test/validation set.

Answer 2

Typically, data augmentation for training convolutional neural networks is only done to the training set. I'm not sure what benefit augmenting the test data would achieve as the value of test data is primarily for model selection and evaluation and you're adding noise to your measurement of those quantities.

Answer 3

Complementing the answers, let my add my 2 cents regarding test-time data augmentation.

Data augmentation can be also performed during test-time with the goal of reducing variance. It can be performed by taking the average of the predictions of modified versions of the input image.

Dataset augmentation may be seen as a way of preprocessing the training set only. Dataset augmentation is an excellent way to reduce the generalization error of most computer vision models. A related idea applicable at test time is to show the model many different versions of the same input (for example, the same image cropped at slightly different locations) and have the different instantiations of the model vote to determine the output. This latter idea can be interpreted as an ensemble approach, and it helps to reduce generalization error. (Deep Learning Book, Chapter 12).

It's a very common practice to apply test-time augmentation. AlexNet and ResNet do that with the 10-crop technique (taking patches from the four corners and the center of the original image and also mirroring them). Inception goes further and generate 144 patches instead of only 10. If you check Kaggle and other competitions, most winners also apply test-time augmentation.

I'm the author of a paper on data augmentation (code) in which we experimented with training and testing augmentation for skin lesion classification (a low-data task). In some cases, using strong data augmentation on training alone is marginally better than not using data augmentation, while using train and test augmentation increases the performance of the model by a very significant margin.