I am training a Convolution Neural Network similar to LeNet5 to detect road signs in the German Traffic Signs Dataset. With about 35,000 training samples I get to 95% validation accuracy. To improve accuracy, I tried augmenting the training set with random Affine transformations of original images.
When I perform augmentation and then normalize my data (mean subtraction and division by standard deviation), the network converges very slowly and validation accuracy actually goes down. However when I perform augmentation after normalization the same network converges much faster and the validation error improves to 97%.
Is there a theoretical basis for this? To me it makes more sense to perform augmentation before normalization because un-normalized images look much better and I can tell if augmentation is working correctly by plotting those images.