Is a Residual Network a Feedforward Network? I am really confused about people comparing feedforward networks to residual networks. This is done in several papers I have read into (just one for example, first line in this paper: https://arxiv.org/pdf/2002.06262.pdf).
A feedforward network is just defined as not having cycles. Residual Networks do not have cycles. So ResNet should be a feedforward neural network, right?
Is this comparasion just done to compare residual networks to traditional feedforward networks or do I miss something?
 A: Neural networks with skip-layer connections have been around for a long time.  A ResNet is indeed a particular type of feed-forward network.
For example, this paper
Peter M. Williams, "Bayesian Regularization and Pruning Using a Laplace Prior", Neural Computation (1995) 7 (1): 117–143. (doi)
mentions skip-layer connections

which suggests it was a fairly standard practice back then (my neural network library at that time included skip-layer connections).  Be interesting to know where the idea was first introduced (CascadeCorrelation perhaps)?
I suspect the key thing here is that very deep networks have a problem with gradient descent optimisation because the gradients can become very small and diffuse as the errors are propagated back from layer to layer.  Skip layer connections helps these gradients to propagate a bit further, so gradient descent remains a bit more effective.  This is not the reason skip-layer connections were used back in the 1990s though, which is that often you can get a more compact model if you have skip-layer connections and don't waste hidden units reproducing the linear-ish components of the required mapping.
