I have 2 questions...
In the paper Deep Residual Learning for Image Recognition, it says
We show by experiments (Fig. 7) that the learned residual functions in general have small responses, suggesting that identity mappings provide reasonable preconditioning.
Q1) What is the relation between reasonable precondition and relatively small deviation of response? Is small deviation good in general? Any kind answers with example would be much appreciated !
Q2) Many internet articles say the point of resnet is addressing the vanishing problem, but the author of resnet clearly showed that the difficulty in deep model is not caused by vanishing gradients.
... The degradation problem suggests that the solvers might have difficulties in approximating identity mappings by multiple nonlinear layers. ... We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with BN [16], which ensures forward propagated signals to have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN. So neither forward nor backward signals vanish.
It seems to me that the resnet have somehow improved the solver, but not related to the vanishing problem. Is that correct?