Is it necessary to skip two or more layers in ResNets?

Question

I was watching Andrew Ng's video about ResNets and in that video he skipped 2 layers while "short cutting" them. For example, if we have input $a^{l}$, next layer will be $a^{l+1} = g(z^{l+1})$ where $z^{l+1} = W^{l+1}a^{l} + b^{l+1}$. Then we have second layer $a^{l+2} = g(z^{l+2})$ where $z^{l+2} = W^{l+2}a^{l+1} + b^{l+2}$. So in the end he writes: $a^{l+2} = g(z^{l+2} + a^{l})$. That is residual block in a nutshell. That is perfectly clear.

But what is not clear to me, why not "short cut" it right after (or better said, in) the first layer, i.e. $a^{l+1} = g(z^{l+1} + a^{l})$? If $W^{l+1}$ and $b^{l+1}$ go to $0$, it would still be $a^{l+1} = g(a^{l})$ so signal would still flow. Why is it necessary to skip more?

Then I read this answer, it says:

The skip connections allow information to skip layers, so, in the forward pass, information from layer l can directly be fed into layer $l+t$ (i.e. the activations of layer $l$ are added to the activations of layer $l+t$), for $t≥2$, and, during the forward pass, the gradients can also flow unchanged from layer $l+t$ to layer $l$.

Which again says $t≥2$, so basically $a^{l+2} = g(z^{l+2} + a^{l})$.

Then I read original paper which says:

The form of the residual function $F$ is flexible. Experiments in this paper involve a function $F$ that has two or three layers (Fig. 5), while more layers are possible. But if $F$ has only a single layer, Eqn.(1) is similar to a linear layer: $y=W_1{x}+x$, for which we have not observed advantages.

Are they implying that in the case of using ReLU as activation function $W_1{x}+x \approx ReLU(W_1{x}+x)$ and that is the reason they skip 2 layers, not one? Because ReLU is almost linear. Or is it just empirical thing "we tried it, it doesn't work, don't do it"?

Would this still hold if one uses sigmoid function, for example? Then $a^{l+1} = g(z^{l+1} + a^{l})$ would have sense maybe?

Answer 1

In the original paper, they say

The function $\mathcal F(x, \{W_i\})$ represents the residual mapping to be learned. For the example in Fig. 2 that has two layers, $\mathcal F = W_2\sigma(W_1x)$ in which $\sigma$ denotes RELU ... We adopt the second nonlinearity after the addition

According to their explanation, the two-layer has only one activation, and therefore one layer would correspond to $W_1x$ only. It's not because $x\approx RELU(x)$. It is linear.

For Andrew Ng's video, if you'd apply the residual connection just after the first layer, the equation would look like the following:

$$a^{l+1}=g(z^{l+1} + a^l)=g(W^{l+1}a^l+b^{l+1}+a^l)=g((I+W^{l+1})a^l+b^{l+1})$$

Which is just learning a weight matrix, but while multiplying $W+I$ with the input instead of $W$, which is why they say it's close to linear. However, I think it can be deduced that the authors have actually tried this as well, but they didn't see any noticeable benefits.

Note: in case of $W$ having less neurons than the dimensions, the residual connection needs to be transformed linearly as well (as described in the original paper), i.e. $W_r a^l$. In this particular case, the summation would be $g((W_r+W^{l+1})a^l+b^{l+1})$, which is learning two matrices $W_r$ and $W^{l+1}$ instead of learning their sum. This doesn't make any contributions to the learning objective.