My understanding of residual layers is that we can take a layer and branch it into two paths. One going the typical path for a network, and the other is an identity mapping forward over the calculating layers. This solves the vanishing gradient problem when we have a lot of layers since regardless of what happens we'll always have the original inputs.
But how doesn't this affect the learning process? If we have
h(x) = Conv(x) -> BN(x) -> ReLU(x), normally this would result in
f(x) = h(x). But in a residual layer we have
f(x) = h(x) + x. How do we stop that additional
x from affecting any of the results?