Can a residual neural network be interpreted as a form of ensemble learning?

Question

I stumbled over this paper: "Veit, A., Wilber, M. & Belongie, S. Residual Networks Behave Like Ensembles of Relatively Shallow Networks. (2016).". In it, they argue that residual neural networks are in fact using short-paths during optimization. Meaning that very deep networks are behaving like multiple not-so-deep networks. These paths apparently show ensemble-like behavior in the sense that they do not strongly depend on each other.

I interpret this as a form of ensemble learning in which we mix predictions from various algorithms to get better results.

Is this intuition of mine correct? And is this, in fact, the reason that ResNets work so well?

We should be able to verify this by training multiple shallow networks and combine their results. Has anyone tried this?

Answer 1

Probably not.

Although resnets have ensemble-like behaviour, it's still a bit of a leap to say they are directly equivalent to an ensemble of shallow networks.

It's quite computationally infeasible since a resnet with only 50 layers has the ensemble equivalent of $2^{50}$ shallow networks.

Also, the "ensemble" of paths in a resnet all share the same common set of weights, whereas a traditional ensemble would learn $2^{50}$ independent sets of weights, so there is a qualitative difference.

Of course if you tied all the weights the right ways you would just end up with a very computationally expensive equivalent of the original network.