I am using a code I altered for sound event classification. The original code, first iterated through all training examples (large chunk), gathered the mean and standard deviation, then normalized all data. After that preprocessing step, the NN would begin training from the normalized saved data.
After my alteration, as well as repacing to another dataset, I have no choice but to skip preprocessing steps. It was pointed out to me by a colleague, that I can simply normalize each batch with respect to itself disregarding other batches. I chose to implement this (in Keras) by adding a batch normalization layer as the first layer.
Note - the training samples are shuffled with each epoch while the validation samples are not.
The result is that the network does not converge, meaning, the error does not droop for nither training nor validation. I do not believe the problem is the dataset, as both datasets are of the same semantic problem and have no technical difference.
Please answer both these question if possible:
- Is it possible to train a network by normalizing each batch only with respect to that batch? What are the pros and cons of such an approach?
- Does a batch normalization layer good enough to do this job or do I have to use other normalization methods?
