1
$\begingroup$

From: Wikipedia on Residual learning in ANNs

"The intuition on why this works is that the neural network collapses into fewer layers in the initial phase, which makes it easier to learn, and thus gradually expands the layers as it learns more of the feature space. During later learning, when all layers are expanded, it will stay closer to the manifold and thus learn faster. A neural network without residual parts will explore more of the feature space. This makes it more vulnerable to small perturbations that cause it to leave the manifold altogether, and require extra training data to get back on track."

I don't not understand

The intuition on why this works is that the neural network collapses into fewer layers in the initial phase, which makes it easier to learn, and thus gradually expands the layers as it learns more of the feature space

What is he talking about collapsing earlier and expanding later? This is a static network. Are the saying the learning is initially more towards simpler features so the skip connections carry most of the information and as the training goes on more features are learnt through the later actual layers? What exactly is the learning progression described here?

During later learning, when all layers are expanded, it will stay closer to the manifold and thus learn faster. A neural network without residual parts will explore more of the feature space. This makes it more vulnerable to small perturbations that cause it to leave the manifold altogether, and require extra training data to get back on track."

What is meant by 'manifold' in this context? What is he saying about feature space and perturbations?

$\endgroup$
1
$\begingroup$

It's probably a better idea to read the original research on why residual connections help rather than a wikipedia page -- I don't think there is really full consensus on residual networks, and that particular page was written almost entirely by one person. The page also doesn't include the mainstream research on why resnets work such as shattered gradients and resnets as an ensemble. Nonetheless, here is my best attempt at an interpretation.

  1. When training a residual network, all but the first few layers of the network could have nearly zero output (weights are often initialized to quite small values), meaning that they add no residual change to the critical path. Then effectively, the network is only a few layers deep and can train quickly. Later, when the network is better trained and the gradient has lower variance, the latter layers may be able to add some useful residual information, and the capacity is effectively "expanded".

  2. I'm also not sure about the use of the term "manifold" here. A common interpretation is that the data lies on some manifold in high dimensional space. As the data travels through each layer in the network, the manifold gets stretched and warped in various ways, with the end goal being to bend the manifold into a form where the data is linearly separable. In that sense, each residual block in a residual network only makes a relatively small distortion to the manifold by its nature of being residual. It's unclear to me what the author meant by "leaving" the manifold.

$\endgroup$
  • $\begingroup$ Thanks! I am interested in this manifold perspective you just described. Can you link me or tell me where I can read a complete description of it? $\endgroup$ – Rahul Deora Oct 22 '18 at 3:27
  • $\begingroup$ @RahulDeora here is a relevant paper $\endgroup$ – shimao Oct 22 '18 at 3:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.