I stumbled over this paper: "Veit, A., Wilber, M. & Belongie, S. Residual Networks Behave Like Ensembles of Relatively Shallow Networks. (2016).". In it, they argue that residual neural networks are in fact using short-paths during optimization. Meaning that very deep networks are behaving like multiple not-so-deep networks. These paths apparently show ensemble-like behavior in the sense that they do not strongly depend on each other.
I interpret this as a form of ensemble learning in which we mix predictions from various algorithms to get better results.
Is this intuition of mine correct? And is this, in fact, the reason that ResNets work so well?
We should be able to verify this by training multiple shallow networks and combine their results. Has anyone tried this?