0
$\begingroup$

My understanding of residual layers is that we can take a layer and branch it into two paths. One going the typical path for a network, and the other is an identity mapping forward over the calculating layers. This solves the vanishing gradient problem when we have a lot of layers since regardless of what happens we'll always have the original inputs.

But how doesn't this affect the learning process? If we have h(x) = Conv(x) -> BN(x) -> ReLU(x), normally this would result in f(x) = h(x). But in a residual layer we have f(x) = h(x) + x. How do we stop that additional x from affecting any of the results?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You want it to affect the results. The simple answer is that it's like building a statistical model: you use linear combination+ nonlinear terms. Note also that a linear deep network has non convex optimization surface see https://arxiv.org/abs/1312.6120

Improve this answer
$\endgroup$
2
  • $\begingroup$ I was under the impression that you can plug these residual blocks into an already trained network and it'll still work. Is that correct or do the weights have to be re-learned? $\endgroup$
    – Matthew Kerian
    Commented Aug 3, 2019 at 19:28
  • $\begingroup$ I don't know where you got that idea. People have trained 100 layer residual networks which would be impossible without. $\endgroup$
    – seanv507
    Commented Aug 3, 2019 at 21:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.