Model Training accuracy decreases and stops learning after applying Tensorflow's batch normalization So I am working on making a model to classify the German traffic signs dataset and everything is fun and games. I build my model of 3 convolutional layers, 2 fully-connected(dense) layers, and 1 last fully-connected(linear) as the logits using tensorflow, I added dropout with keep prob. of 0.5 to the fully-connected layers and Relu activate for all but the the maxpooling and last fully-connected(logits) layer. 
conv Relu + maxpool > conv Relu + maxpool > conv  Relu + maxpool > dense + Relu > dense + Relu > dense(logits)
This model with deep filters for the convolutional layers works great with validation accuracy as high as 97% and training accuracy thats close to 100%, so as a good practice I decided to add batch normalization for the benefits it yields; However, when I add tf.layers.batch_normalization() anywhere with any order my training accuracy dips immediately below the validation accuracy and the model seizes to learn.

Here is the code that reproduces the issue:

 A: I cannot tell the answer from looking at your code and I do not know the structure of the data set and its labels.  However, I can tell  you  what questions I would ask myself when faced with this situation:


*

*What are the size of the batches relative to the number of samples and categories?

*Given that there is very high accuracy to start with.  What does it take to degrade the accuracy - for example how does decimation of the in-sample training data set affect the performance against the final held-back out of sample test set?  


*

*Specifically, assuming the distribution of categories remains proportional, how does performance change when you cut the training data in half (totally omit half the training data), keeping the validation set the same?  And by keeping only 1/4 of the training data? 1/8?    

*How does this compare to degradation where some categories are disproportionately removed so that the implied prior is changed?


*What benefits did  you  expect batch normalization to yield?  What makes it a suitable treatment for this data set?  In other words, how does the change in structure of data presentation effect a change in network weights?   Is there a way that mechanism might harm performance for some problems?

*It appears that in both cases, most of the action is taking place in the first 10 epochs.  In particular, if you are plotting validation accuracy, batch norm seems to follow the characteristic curve of best fit followed by overfitting that is typical of learning algorithms - perhaps to an exaggerated degree, then again, the original data does not have a "peaking" generalization.  Can you apply visualization tools in these early iterations that help understand the dynamics behind those curves?   

*Under what conditions would training accuracy be less than validation accuracy?  In other words, how might having observed a group of samples and their labels make them more likely to be misclassified than samples that were never observed?

*What is the probability of correct classification of data in the training set by random guessing?  And what are the priors (i.e. pcc of always guessing one category, for each category)?  What are those numbers for the training set?


I think that if you can test these questions, it will point you to where the problem is.
