Residual learning approach + manifold From: Wikipedia on Residual learning in ANNs

"The intuition on why this works is that the neural network collapses into fewer layers in the initial phase, which makes it easier to learn, and thus gradually expands the layers as it learns more of the feature space. During later learning, when all layers are expanded, it will stay closer to the manifold and thus learn faster. A neural network without residual parts will explore more of the feature space. This makes it more vulnerable to small perturbations that cause it to leave the manifold altogether, and require extra training data to get back on track."

I don't not understand 

The intuition on why this works is that the neural network collapses into fewer layers in the initial phase, which makes it easier to learn, and thus gradually expands the layers as it learns more of the feature space

What is he talking about collapsing earlier and expanding later? This is a static network. Are the saying the learning is initially more towards simpler features so the skip connections carry most of the information and as the training goes on more features are learnt through the later actual layers? What exactly is the learning progression described here?

During later learning, when all layers are expanded, it will stay closer to the manifold and thus learn faster. A neural network without residual parts will explore more of the feature space. This makes it more vulnerable to small perturbations that cause it to leave the manifold altogether, and require extra training data to get back on track."

What is meant by 'manifold' in this context? What is he saying about feature space and perturbations? 
 A: It's probably a better idea to read the original research on why residual connections help rather than a wikipedia page -- I don't think there is really full consensus on residual networks, and that particular page was written almost entirely by one person. The page also doesn't include the mainstream research on why resnets work such as shattered gradients and resnets as an ensemble. Nonetheless, here is my best attempt at an interpretation.


*

*When training a residual network, all but the first few layers of the network could have nearly zero output (weights are often initialized to quite small values), meaning that they add no residual change to the critical path. Then effectively, the network is only a few layers deep and can train quickly. Later, when the network is better trained and the gradient has lower variance, the latter layers may be able to add some useful residual information, and the capacity is effectively "expanded". 

*I'm also not sure about the use of the term "manifold" here. A common interpretation is that the data lies on some manifold in high dimensional space. As the data travels through each layer in the network, the manifold gets stretched and warped in various ways, with the end goal being to bend the manifold into a form where the data is linearly separable. In that sense, each residual block in a residual network only makes a relatively small distortion to the manifold by its nature of being residual. It's unclear to me what the author meant by "leaving" the manifold.
