Residual Network: Is small deviation of layer response good? What's the point of resnet?

Question

I have 2 questions...

In the paper Deep Residual Learning for Image Recognition, it says

We show by experiments (Fig. 7) that the learned residual functions in general have small responses, suggesting that identity mappings provide reasonable preconditioning.

Q1) What is the relation between reasonable precondition and relatively small deviation of response? Is small deviation good in general? Any kind answers with example would be much appreciated !

Q2) Many internet articles say the point of resnet is addressing the vanishing problem, but the author of resnet clearly showed that the difficulty in deep model is not caused by vanishing gradients.

... The degradation problem suggests that the solvers might have difficulties in approximating identity mappings by multiple nonlinear layers. ... We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with BN [16], which ensures forward propagated signals to have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN. So neither forward nor backward signals vanish.

It seems to me that the resnet have somehow improved the solver, but not related to the vanishing problem. Is that correct?

Answer 1

Both questions are obviously related.

2) Let me start with the second question: the authors acknowledge that vanishing gradients is one of the problems, but that this one is addressed by the normalization layer. Rather, the problem addressed by resnet is that, with increasing depth of the network, they observe an increased training error (which is not due to overfitting).

What they argue it is is easier to incrementaly learn the residuals than to learn full mappings: given one representation of the input, take the "error", that is, how much it deviates from the full input and improve on it (see section "Residual Representations" for details). And precisely that seems to be the cause of the higher training error with increasing depth of a standard deep network network (with no shortcut connections).

1) Going back to the first one: a small deviation is good in general in order to avoid saturation and thus vanishing gradients. This topic is well addressed in the paper Efficient Backprop, by LeCunn et al. Even though an old paper, but many of the motivations are well described there.

The idea of resnet is that the input from one layer to the next is better preconditioning reflects in a system that is able, with less parameters (they compare against VGG), to optimize a network with way more layers, providing a better generalization. In loose way, with better preconditining, you learn better, more accurate represenations of the input.