I have a few questions about the paper Deep Residual Learning for Image Recognition by Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun.
The building blocks of residual networks can be viewed as follows: data passed to the right branch, convolution, scaling, convolution; then in the right branch: identity mapping or convolution; and after that, both branches' outputs are summed.
Why does this allow the training of deep networks, escaping network saturation at deep levels? I didn't get the idea from the paper. Is this summation a reminder of what happened a few layers ago, a reference point? Or is it just clever regularization?
How was the amount of right branch layers chosen?
Why do we train scale layer on the right branch, according to this caffe architecture?