36
$\begingroup$

I am interested in regression with neural networks.

Neural networks with zero hidden nodes + skip-layer connections are linear models.

What about the same neural nets but with hidden nodes ? I am wondering what would be the role of the skip-layer connections ?

Intuitively, i would say that if you include the skip-layer connections, then the final model will a sum of a linear model + some non-linear parts.

Is there any advantage or disadvantage in adding skip-layer connections to neural nets ?

$\endgroup$
51
$\begingroup$

I am very late to the game, but I wanted to post to reflect some current developments in convolutional neural networks with respect to skip connections.

A Microsoft Research team recently won the ImageNet 2015 competition and released a technical report Deep Residual Learning for Image Recognition describing some of their main ideas.

One of their main contributions is this concept of deep residual layers. These deep residual layers use skip connections. Using these deep residual layers, they were able to train a 152 layer conv net for ImageNet 2015. They even trained a 1000+ layer conv net for the CIFAR-10.

The problem that motivated them is the following:

When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error...

The idea is if that if you take a "shallow" network and just stack on more layers to create a deeper network, the performance of the deeper network should be at least as good as the shallow network as the deeper network could learn the exact shallow network by setting the new stacked layers to identity layers (in reality we know this is probably highly unlikely to happen using no architectural priors or current optimization methods). They observed that this was not the case and that training error sometimes got worse when they stacked more layers on top of a shallower model.

So this motivated them to use skip connections and use so-called deep residual layers to allow their network to learn deviations from the identity layer, hence the term residual, residual here referring to difference from the identity.

They implement skip connections in the following manner: enter image description here

So they view the map $\mathcal{F}(x) := \mathcal{H}(x) - x$ as some residual map. They use a skip layer connection to cast this mapping into $\mathcal{F}(x) + x = \mathcal{H}(x)$. So if the residual $\mathcal{F}(x)$ is "small", the map $\mathcal{H}(x)$ is roughly the identity.

In this manner the use of deep residual layers via skip connections allows their deep nets to learn approximate identity layers, if that is indeed what is optimal, or locally optimal. Indeed they claim that their residual layers:

We show by experiments (Fig. 7) that the learned residual functions in general have small responses

As to why exactly this works they don't have an exact answer. It is highly unlikely that identity layers are optimal, but they believe that using these residual layers helps precondition the problem and that it's easier to learn a new function given a reference/baseline of comparison to the identity mapping than to learn one "from scratch" without using the identity baseline. Who knows. But I thought this would be a nice answer to your question.

By the way, in hindsight: sashkello's answer is even better isn't it?

$\endgroup$
3
  • $\begingroup$ What do you mean by precondition? (also your answer is much better than sashkello's by a lot imho). Also, so the problem is that adding further layers without the skip connections makes it hard for the deeper network to even find the identity mapping? (for some reason?) $\endgroup$ Nov 20 '16 at 4:31
  • 1
    $\begingroup$ This is a great answer, thank you. Just wanted to update that there is a great paper, arxiv.org/pdf/1701.09175.pdf, that investigates why this actually improves learning. $\endgroup$ Jan 26 '20 at 13:52
  • $\begingroup$ Having a skip layer connection will not allow using Adam optimizer. Is that true ?? @IndieAl $\endgroup$ Sep 28 at 18:49
28
$\begingroup$

In theory, skip-layer connections should not improve on the network performance. But, since complex networks are hard to train and easy to overfit it may be very useful to explicitly add this as a linear regression term, when you know that your data has a strong linear component. This hints the model in a right direction... In addition, this is more interpretable since it presents your model as linear + perturbations, unraveling a bit of a structure behind the network, which is usually seen merely as a black box.

$\endgroup$
4
  • 1
    $\begingroup$ @sashkello can you elaborate on ' skip-layer connections should not improve on the network performance', in theoretical way. That would be great to understand. $\endgroup$
    – hunch
    Dec 17 '16 at 5:11
  • 1
    $\begingroup$ @hunch It's because you can model this without the need of skip-layer. Skip-layer is just a linear term. Sigmoid function is ~linear around 0. So, a skip-layer connection will behave like a hidden node with very small input weights. $\endgroup$
    – sashkello
    Dec 18 '16 at 22:17
  • 1
    $\begingroup$ Technically, the skip connection is the identity function, not a linear. So it is identity + perturbations. The activation function could give a linear transformation, but with any activation you use these days they are nonlinear. The important idea with skip connections is so that you can increase network depth without hurting performance; e.g. if first layers converge on solution, it can easily learn to ignore remaining layers. $\endgroup$
    – Azmisov
    Jun 24 at 20:45
  • $\begingroup$ @Azmisov Your comment cleared a lot of doubts in my mind. Thanks! $\endgroup$
    – Rishab P.
    Oct 21 at 10:23
6
$\begingroup$

My old neural network toolbox (I mostly use kernel machines these days) used L1 regularisation to prune away redundant weights and hidden units, and also had skip-layer connections. This has the advantage that if the problem is essentially linear, the hidden units tend to get pruned and you are left with a linear model, which clearly tells you that the problem is linear.

As sashkello (+1) suggests, MLPs are universal approximators, so skip layer connections won't improve results in the limit of infinite data and an infinite number of hidden units (but when do we ever approach that limit?). The real advantage is that it makes estimating good values for the weights easier if the network architecture is well matched to the problem, and you may be able to use a smaller network and obtain better generalisation performance.

However, as with most neural network questions, generally the only way to find out if it will be helpful or harmful for a particular dataset is to try it and see (using a reliable performance evaluation procedure).

$\endgroup$
1
  • 3
    $\begingroup$ "try it and see" - golden words :) $\endgroup$
    – sashkello
    Apr 23 '13 at 22:52
2
$\begingroup$

An in-depth explanation of skip connections from multiple perspectives can be found here:

https://theaisummer.com/skip-connections/

Here I provide the main point from the article:

Basically, skip connection is a standard module in many convolutional architectures. By using a skip connection, we provide an alternative path for the gradient (with backpropagation). It is experimentally validated that these additional paths are often beneficial for the model convergence. Skip connections in deep architectures, as the name suggests, skip some layer in the neural network and feeds the output of one layer as the input to the next layers (instead of only the next one).

As previously explained, using the chain rule, we must keep multiplying terms with the error gradient as we go backwards. However, in the long chain of multiplication, if we multiply many things together that are less than one, then the resulting gradient will be very small. Thus, the gradient becomes very small as we approach the earlier layers in a deep architecture. In some cases, the gradient becomes zero, meaning that we do not update the early layers at all.

In general, there are two fundamental ways that one could use skip connections through different non-sequential layers:

a) addition as in residual architectures,

b) concatenation as in densely connected architectures.

I hope it clarifies your understanding!

$\endgroup$
0
$\begingroup$

Based on Bishop 5.1. Feed-forward Network Functions:

[A way to generalize] of the network architecture is to include skip-layer connections, each of which is associated with a corresponding adaptive parameter. For instance, in a two-layer network these would go directly from inputs to outputs. In principle, a network with sigmoidal hidden units can always mimic skip layer connections (for bounded input values) by using a sufficiently small first-layer weight that, over its operating range, the hidden unit is effectively linear, and then compensating with a large weight value from the hidden unit to the output. In practice, however, it may be advantageous to include skip-layer connections explicitly.enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.