I was going through the cifar10 tutorial from tensorflow and was trying to understand how I can compare different learning algorithms and models with standard benchmarks in the context of data set augmentations.

In vision its common to crop and apply further transformations to data sets as data augmentation. If the community does not use the exact same set of augmentation I would assume that everyone would have a slightly different data set, which would make scientific findings a little confusing to interpret clearly (since everyone has a different data set). Since the tensorflow flow tutorial uses on the fly data set augmentation, it made me wonder, if I re-use that code, how can I actually know if my algorithm/model is doing well or if its not doing as well due to the current set of data augmentations. How does the community deal with this and when one says "my accuracy for cifar10 or ImageNet is 97%", what cifar10 or ImageNet data set are they referring to?

To avoid this issue it seems the MNIST data set has a webpage where one can find a fixed data set corresponding to each transformation. These are:

• mnist-rot
• mnist-back-rand
• mnist-back-image
• mnist-rot-back-image

does this mean that if I do data set augmentations, I have to use the ones from some fixed database unless I am doing some sort of novel data set augmentation (and in this special case make my specific data set publicly available)? Or how does the machine learning community deal with this issue of data set augmentation? In a way, I guess I am sort of looking for what the term "benchmark" really means and how researchers in Machine Learning benchmark.

After some more time I thought about it more. It seems to me that the data set augmentation could in theory be thought as a part of the "algorithm" or "method". With this mentality, whatever the model gets for training is legal, in other words, you can do whatever pre-processing you want on the train data (even if this makes it more ambiguous if the actual algorithm or the data set augmentation were responsible for the improvement or worsening). However, what seems to be somewhat important is to keep the test data fixed. If this is not fixed, then its hard to compare with the performance with other training algorithms since everyone would have their own special test set. If this is fixed it would seem that one would bypass this issue of not being consistent across data sets.

However, if one really needs to know the performance on rotations because that is what we are studying, then it seems reasonable to also require some test set with this property (i.e. that its rotated). Though, this seems to be more important if that is the specific goal of the inquiry, otherwise, it would seem reasonable to keep the data set fixed to ease comparison with other researchers/benchmarks.