I am currently experimenting with different settings for a U-Net (https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/) based image segmentation and I was unable to find out if it makes any sense to place a BatchNorm layer after a convolutional layer which includes ReLU. Such as follows:
conv1 = Conv2D(64, 3, activation='relu', padding='same', kernel_initializer=gauss())(inputs) batch1 = BatchNormalization()(conv1) conv1 = Conv2D(64, 3, activation='relu', padding='same', kernel_initializer=gauss(stddev=sqrt(2 / (9 * 64))))(batch1)
Can somebody tell me or point out the relevant literature to explain if it is possible to use BatchNorm like shown above or if it should always be placed before ReLU.
I did check out this discussion on reddit, where some people do seem to recommend using BatchNorm after ReLU: https://www.reddit.com/r/MachineLearning/comments/67gonq/d_batch_normalization_before_or_after_relu/
The U-Net does indeed segment my images with these settings, but not really much better than when using two Dropout layers at the end of the contracting pathway or no Dropout or BatchNorm layers at all.