I was watching Andrew Ng's video about ResNets and in that video he skipped 2 layers while "short cutting" them. For example, if we have input $a^{l}$, next layer will be $a^{l+1} = g(z^{l+1})$ where $z^{l+1} = W^{l+1}a^{l} + b^{l+1}$. Then we have second layer $a^{l+2} = g(z^{l+2})$ where $z^{l+2} = W^{l+2}a^{l+1} + b^{l+2}$. So in the end he writes: $a^{l+2} = g(z^{l+2} + a^{l})$. That is residual block in a nutshell. That is perfectly clear.

But what is not clear to me, why not "short cut" it right after (or better said, in) the first layer, i.e. $a^{l+1} = g(z^{l+1} + a^{l})$? If $W^{l+1}$ and $b^{l+1}$ go to $0$, it would still be $a^{l+1} = g(a^{l})$ so signal would still flow. Why is it necessary to skip more?

Then I read this answer, it says:

The skip connections allow information to skip layers, so, in the forward pass, information from layer l can directly be fed into layer $l+t$ (i.e. the activations of layer $l$ are added to the activations of layer $l+t$), for $t≥2$, and, during the forward pass, the gradients can also flow unchanged from layer $l+t$ to layer $l$.

Which again says $t≥2$, so basically $a^{l+2} = g(z^{l+2} + a^{l})$.

Then I read original paper which says:

The form of the residual function $F$ is flexible. Experiments in this paper involve a function $F$ that has two or three layers (Fig. 5), while more layers are possible. But if $F$ has only a single layer, Eqn.(1) is similar to a linear layer: $y=W_1{x}+x$, for which we have not observed advantages.

Are they implying that in the case of using ReLU as activation function $W_1{x}+x \approx ReLU(W_1{x}+x)$ and that is the reason they skip 2 layers, not one? Because ReLU is almost linear. Or is it just empirical thing "we tried it, it doesn't work, don't do it"?

Would this still hold if one uses sigmoid function, for example? Then $a^{l+1} = g(z^{l+1} + a^{l})$ would have sense maybe?


1 Answer 1


In the original paper, they say

The function $\mathcal F(x, \{W_i\})$ represents the residual mapping to be learned. For the example in Fig. 2 that has two layers, $\mathcal F = W_2\sigma(W_1x)$ in which $\sigma$ denotes RELU ... We adopt the second nonlinearity after the addition

According to their explanation, the two-layer has only one activation, and therefore one layer would correspond to $W_1x$ only. It's not because $x\approx RELU(x)$. It is linear.

For Andrew Ng's video, if you'd apply the residual connection just after the first layer, the equation would look like the following:

$$a^{l+1}=g(z^{l+1} + a^l)=g(W^{l+1}a^l+b^{l+1}+a^l)=g((I+W^{l+1})a^l+b^{l+1})$$

Which is just learning a weight matrix, but while multiplying $W+I$ with the input instead of $W$, which is why they say it's close to linear. However, I think it can be deduced that the authors have actually tried this as well, but they didn't see any noticeable benefits.

Note: in case of $W$ having less neurons than the dimensions, the residual connection needs to be transformed linearly as well (as described in the original paper), i.e. $W_r a^l$. In this particular case, the summation would be $g((W_r+W^{l+1})a^l+b^{l+1})$, which is learning two matrices $W_r$ and $W^{l+1}$ instead of learning their sum. This doesn't make any contributions to the learning objective.

  • $\begingroup$ If I understood right, you are saying that $W^{l+1}a^{l}$ is close to linear because it hasn't passed through activation function yet, but $W^{l+2}a^{l+1}$ in $a^{l+2}$ is nonlinear because $a^{l+1}$ is output of ReLU (while $a^{l}$ is not)? $\endgroup$
    – dosvarog
    Commented Jan 8, 2022 at 14:26
  • $\begingroup$ That’s right @dosvarog $\endgroup$
    – gunes
    Commented Jan 8, 2022 at 14:27
  • $\begingroup$ Okay, thank you, that is clear now. But what if have a normal network and I decide to put just one residual block in the middle of it? Then the input (let's us call it again $a^{l}$ for the sake of argument) to that block would already passed many nonlinearities. Then $W^{l+1}a^{l}$ would not be close to linear. Or would it? What am I missing here? $\endgroup$
    – dosvarog
    Commented Jan 8, 2022 at 14:37
  • 1
    $\begingroup$ Yes, because the input (also mentioned in the papers) is not the network's input but the residual block's input. $\endgroup$
    – gunes
    Commented Jan 10, 2022 at 20:28
  • 1
    $\begingroup$ Yes, this also makes sense because it's added after the activation. It's quite similar to what's being already done except the linear transformation. $\endgroup$
    – gunes
    Commented Jan 10, 2022 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.