I was going through the MNIST example that comes with MatConvNet were they attempt to demonstrate how to use the API of their Batch Normalization. However if you take a look at the training file there is a line in cnn_train.m (in the accumulate_gradients.m function) as follows (I re-formated it to ease readability):

if j == 3 && strcmp(net.layers{l}.type, 'bnorm')
  %% special case for learning bnorm moments
  thisLR = net.layers{l}.learningRate(j) ;
  net.layers{l}.weights{j} = (1-thisLR) * net.layers{l}.weights{j} + (thisLR/batchSize) * res(l).dzdw{j} ;

where usually they would do a gradient descent update but instead do the mysterious above line.

Why do we need such a weird update?

According to the original batch normalization paper 1 there are only two parameters to train, the scale $\gamma^{(k)}$ and the shift $\beta^{(k)}$. However, in that line of code they only do the mysterious line if j==3 suggesting there is a 3rd parameter they update. To further support this claim if you inspect when they initialize the BN layer there is further evidence of the existence of this third parameter:

% --------------------------------------------------------------------
function net = insertBnorm(net, l)
% --------------------------------------------------------------------
assert(isfield(net.layers{l}, 'weights'));
ndim = size(net.layers{l}.weights{1}, 4);
layer = struct('type', 'bnorm', ...
               'weights', {{ones(ndim, 1, 'single'), zeros(ndim, 1, 'single')}}, ...
               'learningRate', [1 1 0.05], ...
               'weightDecay', [0 0]) ;
net.layers{l}.biases = [] ;
net.layers = horzcat(net.layers(1:l), layer, net.layers(l+1:end)) ;

note that even though there are only two matrices/tensors for the variable weights there is a third mysterious parameter for the learning rate which I am trying to figure out what it is.

Does anyone have an idea of what it is?

Furthermore, looking at their library's documentation they have:


which makes me think that the only thing could be a gradient with respect to the input X but that would make sense to me to use as an update rule.

Any idea of whats going on with this third mysterious index/parameter?

After thinking about it more, I have a feeling that it might have to do with updating the moments (since thats what the comment says). I am assuming that it has to do with updating the mean $\mu$ and the variance $\sigma$. However, that doesn't make sense to me because I thought those depended on the population mean/std or the batch mean/std. Anyone can clarify?

Even if one has to update the moments, why does one have to update them with a different rule than the other parameters? If you look closely at that file, the rest of the parameters are updated using momentum but one is not.

1: Ioffe S. and Szegedy C. (2015),
"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift",
Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 2015.
Journal of Machine Learning Research: W&CP volume 37


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.