Can neural networks be categorized via set inclusion or ordering? If we were to consider the classes of:

*

*linear regression

*polynomial regression

*logistic regression

*softmax regresson

*mlp

*conv net

*recurrent net

*residual net...

Does there exist categorization of these classes based on

*

*set inclusion
or

*ordering?

For example, linear regression models are subsets of MLPs. There exist a simple set inclusion relationship between these two classes of models.
A polynomial regression can capture a linear regression
So we have linear reg $\subset$ poly reg $\subset$ MLP...
Does this generalize?
 A: Besides many trivial inclusion properties, there are some more interesting.
First, I would stipulate that feed-forward NNs (FNNs) are NNs where each hidden node is an activation function $a$ of an affine map of the inputs:
$$
y_i = a(\sum w_{ij}x_j + b_i).
$$
Then, MLPs are sometimes considered to be FNNs that have to be fully connected. In this case, of course, they are not anymore a superset of e.g. Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs).
CNNs, RNNs, and Residual Neural Networks (ResNets) can all be considered special cases of FNNs. (ResNets can be made FNNs by concatenating layers with the duplicate of the previous one.)
A vanilla RNN can be considered an FNN (unrolling). Interestingly, although LSTMs are often considered RNNs, they are not FNNs anymore, because of the element-wise multiplication nodes. In this way, LSTMs are fundamentally different.
For the classical regressions, note that often polynomial regression is considered linear regression because it creates a linear combination of the base functions, which in this case are the monomials.
In general, neural networks are often considered non-parametric models, even though they have a finite number of parameters, but since this number is so large, and those parameters mostly don't have any meaning anymore, they are considered nonparametric. And while neural networks might be able to approximate some model-driven methods, they don't properly contain them. And they will not be able to replace them.
