I am using a code I altered for sound event classification. The original code, first iterated through all training examples (large chunk), gathered the mean and standard deviation, then normalized all data. After that preprocessing step, the NN would begin training from the normalized saved data.

After my alteration, as well as repacing to another dataset, I have no choice but to skip preprocessing steps. It was pointed out to me by a colleague, that I can simply normalize each batch with respect to itself disregarding other batches. I chose to implement this (in Keras) by adding a batch normalization layer as the first layer.

Note - the training samples are shuffled with each epoch while the validation samples are not.

The result is that the network does not converge, meaning, the error does not droop for nither training nor validation. I do not believe the problem is the dataset, as both datasets are of the same semantic problem and have no technical difference.

Please answer both these question if possible:

  • Is it possible to train a network by normalizing each batch only with respect to that batch? What are the pros and cons of such an approach?
  • Does a batch normalization layer good enough to do this job or do I have to use other normalization methods?
  • $\begingroup$ Are the batches randomised? It is not that eg one batch has one class of sounds and another batch has different group? $\endgroup$ – seanv507 Oct 1 '19 at 19:54
  • $\begingroup$ Randomized on training but not on validation. What do you mean by eg? $\endgroup$ – havakok Oct 2 '19 at 6:06
  • $\begingroup$ e.g. Meaning for example. When you say not randomised for validation, are you looking at convergence on your validation set? Then won't your mean, variances be wrong...I can't remember how validation data is batch normalised? $\endgroup$ – seanv507 Oct 2 '19 at 6:30
  • 1
    $\begingroup$ I have edited the question with respect to your questions. $\endgroup$ – havakok Oct 2 '19 at 8:48
  • $\begingroup$ Are you able to attach code before, after change? $\endgroup$ – seanv507 Oct 2 '19 at 10:39

What is the batch size, data set size. Maybe you can output the batch means and sds and compare to global. Essentially that determines whether the bn will work the same


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.