The traditional approach to batch normalization is to estimate the mean and variance from the batch and use it to normalize the data at different layers, while keeping moving averages that you should later use at test/prediction time. My question is, wouldn't it be better to use the moving averages at training time too?
Of course, it would be worse at the very beginning, but if you use, for example, an exponential moving average with a small initial decay (you can increase it later) the moving average will be okay after a few mini-batches. And then, if you get a mini-batch that, just by chance, is further than usual from the average, wouldn't you rather train using the same average that you'll have at test time?
The extreme case would obviously be an online learning setup with one example per batch; basically every single example would turn to zero at training time, but not at test time.