What problem does Residual Nets solve that batch normalization does not solve?

Question

I was trying to understand how the contributions of Residual Nets differed from batch normalization. I have read both papers but its still not clear to me.

As far as I can tell batch normalization essentially solved the issue of vanishing and exploding gradients. However, intuitively for me it seems that in principle this was mainly caused by depth of a network. Thus, why is it that batch normalization is not able to train networks that are as deep as residual networks? What is special about ResNets that batch norm does not do?

How do the contributions of ResNets and batch norm differ? Is it that skip connection can learn the identity better and thus much deep nets are trainable? Do ResNets not work without batch norm meaning ResNets do not solve the vanishing/exploding gradient problem at all?

How do the contributions differ?

Answer 1

"Skip connections eliminate singularities" by A. Emin Orhan, Xaq Pitkow offers an explanation: residual connections ameliorate singularities in neural networks.

Skip connections made the training of very deep networks possible and have become an indispensable component in a variety of neural architectures. A completely satisfactory explanation for their success remains elusive. Here, we present a novel explanation for the benefits of skip connections in training very deep networks. The difficulty of training deep networks is partly due to the singularities caused by the non-identifiability of the model. Several such singularities have been identified in previous works: (i) overlap singularities caused by the permutation symmetry of nodes in a given layer, (ii) elimination singularities corresponding to the elimination, i.e. consistent deactivation, of nodes, (iii) singularities generated by the linear dependence of the nodes. These singularities cause degenerate manifolds in the loss landscape that slow down learning. We argue that skip connections eliminate these singularities by breaking the permutation symmetry of nodes, by reducing the possibility of node elimination and by making the nodes less linearly dependent. Moreover, for typical initializations, skip connections move the network away from the “ghosts” of these singularities and sculpt the landscape around them to alleviate the learning slow-down. These hypotheses are supported by evidence from simplified models, as well as from experiments with deep networks trained on real-world datasets.

I'm not aware of a layer or batch normalization strategy that can accomplish this.

Answer 2

Batch normalization mitigates issues such as:

• how can you use a large learning rate without 'breaking' the network, eg accidentally kill all ReLUs, push the preceeding layer so far in one direction, that their output is always negative, and thus no gradients flow
• avoid having to shift all weights in one direction by a constant amounts, or multiply by a constant magnitude, when you can simply learn two single scalars to handle this

However, vanishing gradients are typically considered to be caused in a large part by the passage through non-linear 'activation' functions, such as sigmoid or tanh. Using ReLU mitigates this issue to a large extend, a large part of the reason for the widespread usage of ReLU, but doesn't solve vanishing gradients entirely.

Using residual nets gives a 'bus' that all gradients can flow down, directly, albeit modified somewhat by the attached residual blocks.

I put a diagram of how I see a residual net working in another answer, to a similar-ish question, at Gradient backpropagation through ResNet skip connections

Answer 3

I think the main reason in that deep networks forget the input data from some time thus their behavior depends from previous layers. And it the same as a broken phone. Resid connections adding some hopping over few layers, which gives better understanding what happening.