Neural network with skip-layer connections

I am interested in regression with neural networks.

Neural networks with zero hidden nodes + skip-layer connections are linear models.

What about the same neural nets but with hidden nodes ? I am wondering what would be the role of the skip-layer connections ?

Intuitively, i would say that if you include the skip-layer connections, then the final model will a sum of a linear model + some non-linear parts.

Is there any advantage or disadvantage in adding skip-layer connections to neural nets ?

I am very late to the game, but I wanted to post to reflect some current developments in convolutional neural networks with respect to skip connections.

A Microsoft Research team recently won the ImageNet 2015 competition and released a technical report Deep Residual Learning for Image Recognition describing some of their main ideas.

One of their main contributions is this concept of deep residual layers. These deep residual layers use skip connections. Using these deep residual layers, they were able to train a 152 layer conv net for ImageNet 2015. They even trained a 1000+ layer conv net for the CIFAR-10.

The problem that motivated them is the following:

When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error...

The idea is if that if you take a "shallow" network and just stack on more layers to create a deeper network, the performance of the deeper network should be at least as good as the shallow network as the deeper network could learn the exact shallow network by setting the new stacked layers to identity layers (in reality we know this is probably highly unlikely to happen using no architectural priors or current optimization methods). They observed that this was not the case and that training error sometimes got worse when they stacked more layers on top of a shallower model.

So this motivated them to use skip connections and use so-called deep residual layers to allow their network to learn deviations from the identity layer, hence the term residual, residual here referring to difference from the identity.

They implement skip connections in the following manner:

So they view the map $\mathcal{F}(x) := \mathcal{H}(x) - x$ as some residual map. They use a skip layer connection to cast this mapping into $\mathcal{F}(x) + x = \mathcal{H}(x)$. So if the residual $\mathcal{F}(x)$ is "small", the map $\mathcal{H}(x)$ is roughly the identity.

In this manner the use of deep residual layers via skip connections allows their deep nets to learn approximate identity layers, if that is indeed what is optimal, or locally optimal. Indeed they claim that their residual layers:

We show by experiments (Fig. 7) that the learned residual functions in general have small responses

As to why exactly this works they don't have an exact answer. It is highly unlikely that identity layers are optimal, but they believe that using these residual layers helps precondition the problem and that it's easier to learn a new function given a reference/baseline of comparison to the identity mapping than to learn one "from scratch" without using the identity baseline. Who knows. But I thought this would be a nice answer to your question.

By the way, in hindsight: sashkello's answer is even better isn't it?

• What do you mean by precondition? (also your answer is much better than sashkello's by a lot imho). Also, so the problem is that adding further layers without the skip connections makes it hard for the deeper network to even find the identity mapping? (for some reason?) – Charlie Parker Nov 20 '16 at 4:31

In theory, skip-layer connections should not improve on the network performance. But, since complex networks are hard to train and easy to overfit it may be very useful to explicitly add this as a linear regression term, when you know that your data has a strong linear component. This hints the model in a right direction... In addition, this is more interpretable since it presents your model as linear + perturbations, unraveling a bit of a structure behind the network, which is usually seen merely as a black box.

• @sashkello can you elaborate on ' skip-layer connections should not improve on the network performance', in theoretical way. That would be great to understand. – hunch Dec 17 '16 at 5:11
• @hunch It's because you can model this without the need of skip-layer. Skip-layer is just a linear term. Sigmoid function is ~linear around 0. So, a skip-layer connection will behave like a hidden node with very small input weights. – sashkello Dec 18 '16 at 22:17

My old neural network toolbox (I mostly use kernel machines these days) used L1 regularisation to prune away redundant weights and hidden units, and also had skip-layer connections. This has the advantage that if the problem is essentially linear, the hidden units tend to get pruned and you are left with a linear model, which clearly tells you that the problem is linear.

As sashkello (+1) suggests, MLPs are universal approximators, so skip layer connections won't improve results in the limit of infinite data and an infinite number of hidden units (but when do we ever approach that limit?). The real advantage is that it makes estimating good values for the weights easier if the network architecture is well matched to the problem, and you may be able to use a smaller network and obtain better generalisation performance.

However, as with most neural network questions, generally the only way to find out if it will be helpful or harmful for a particular dataset is to try it and see (using a reliable performance evaluation procedure).

• "try it and see" - golden words :) – sashkello Apr 23 '13 at 22:52