I am interested in regression with neural networks.
Neural networks with zero hidden nodes + skip-layer connections are linear models.
What about the same neural nets but with hidden nodes ? I am wondering what would be the role of the skip-layer connections ?
Intuitively, i would say that if you include the skip-layer connections, then the final model will a sum of a linear model + some non-linear parts.
Is there any advantage or disadvantage in adding skip-layer connections to neural nets ?
I am very late to the game, but I wanted to post to reflect some current developments in convolutional neural networks with respect to skip connections.
A Microsoft Research team recently won the ImageNet 2015 competition and released a technical report Deep Residual Learning for Image Recognition describing some of their main ideas.
One of their main contributions is this concept of deep residual layers. These deep residual layers use skip connections. Using these deep residual layers, they were able to train a 152 layer conv net for ImageNet 2015. They even trained a 1000+ layer conv net for the CIFAR-10.
The problem that motivated them is the following:
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error...
The idea is if that if you take a "shallow" network and just stack on more layers to create a deeper network, the performance of the deeper network should be at least as good as the shallow network as the deeper network could learn the exact shallow network by setting the new stacked layers to identity layers (in reality we know this is probably highly unlikely to happen using no architectural priors or current optimization methods). They observed that this was not the case and that training error sometimes got worse when they stacked more layers on top of a shallower model.
So this motivated them to use skip connections and use so-called deep residual layers to allow their network to learn deviations from the identity layer, hence the term residual, residual here referring to difference from the identity.
They implement skip connections in the following manner:
So they view the map $\mathcal{F}(x) := \mathcal{H}(x) - x$ as some residual map. They use a skip layer connection to cast this mapping into $\mathcal{F}(x) + x = \mathcal{H}(x)$. So if the residual $\mathcal{F}(x)$ is "small", the map $\mathcal{H}(x)$ is roughly the identity.
In this manner the use of deep residual layers via skip connections allows their deep nets to learn approximate identity layers, if that is indeed what is optimal, or locally optimal. Indeed they claim that their residual layers:
We show by experiments (Fig. 7) that the learned residual functions in general have small responses
As to why exactly this works they don't have an exact answer. It is highly unlikely that identity layers are optimal, but they believe that using these residual layers helps precondition the problem and that it's easier to learn a new function given a reference/baseline of comparison to the identity mapping than to learn one "from scratch" without using the identity baseline. Who knows. But I thought this would be a nice answer to your question.
By the way, in hindsight: sashkello's answer is even better isn't it?
In theory, skip-layer connections should not improve on the network performance. But, since complex networks are hard to train and easy to overfit it may be very useful to explicitly add this as a linear regression term, when you know that your data has a strong linear component. This hints the model in a right direction... In addition, this is more interpretable since it presents your model as linear + perturbations, unraveling a bit of a structure behind the network, which is usually seen merely as a black box.
My old neural network toolbox (I mostly use kernel machines these days) used L1 regularisation to prune away redundant weights and hidden units, and also had skip-layer connections. This has the advantage that if the problem is essentially linear, the hidden units tend to get pruned and you are left with a linear model, which clearly tells you that the problem is linear.
As sashkello (+1) suggests, MLPs are universal approximators, so skip layer connections won't improve results in the limit of infinite data and an infinite number of hidden units (but when do we ever approach that limit?). The real advantage is that it makes estimating good values for the weights easier if the network architecture is well matched to the problem, and you may be able to use a smaller network and obtain better generalisation performance.
However, as with most neural network questions, generally the only way to find out if it will be helpful or harmful for a particular dataset is to try it and see (using a reliable performance evaluation procedure).
An in-depth explanation of skip connections from multiple perspectives can be found here:
https://theaisummer.com/skip-connections/
Here I provide the main point from the article:
Basically, skip connection is a standard module in many convolutional architectures. By using a skip connection, we provide an alternative path for the gradient (with backpropagation). It is experimentally validated that these additional paths are often beneficial for the model convergence. Skip connections in deep architectures, as the name suggests, skip some layer in the neural network and feeds the output of one layer as the input to the next layers (instead of only the next one).
As previously explained, using the chain rule, we must keep multiplying terms with the error gradient as we go backwards. However, in the long chain of multiplication, if we multiply many things together that are less than one, then the resulting gradient will be very small. Thus, the gradient becomes very small as we approach the earlier layers in a deep architecture. In some cases, the gradient becomes zero, meaning that we do not update the early layers at all.
In general, there are two fundamental ways that one could use skip connections through different non-sequential layers:
a) addition as in residual architectures,
b) concatenation as in densely connected architectures.
I hope it clarifies your understanding!
Based on Bishop 5.1. Feed-forward Network Functions:
[A way to generalize] of the network architecture is to include skip-layer connections, each of which is associated with a corresponding adaptive parameter. For instance, in a two-layer network these would go directly from inputs to outputs. In principle, a network with sigmoidal hidden units can always mimic skip layer connections (for bounded input values) by using a sufficiently small first-layer weight that, over its operating range, the hidden unit is effectively linear, and then compensating with a large weight value from the hidden unit to the output. In practice, however, it may be advantageous to include skip-layer connections explicitly.
