As we delve into Neural NetworksNeural Networks literature, we get to identify other methods with neuromorphic topologies ("Neural-Network"-like architectures). And I'm not talking about the Universal Approximation TheoremUniversal Approximation Theorem. Examples are given below.

Then, it makes me wonder: what is the definition of an artificial Neural Network? Its topology appears to cover everything.

Examples:

One of the first identifications we make is between PCA and a linear Autoencoder with tied-weights in the encoder and decoder and thresholded activations in the bottleneck layer.

Also, a common identification is done between linear models (logistic regression in special) and a Neural Network with no hidden layer and a single output layer. This identification opens several doors.

Fourier and Taylor series? ANNs. SVMSVM? ANN. Gaussian Process? ANN (with single hidden layer with infinite hidden units).

And so, just as easily, we can incorporate arbitrary regularized versions with specialized loss functions of these algorithms into a Neural Network framework.

But the more we dig, the more similarities appear. I just stumbled into Deep Neural Decision Trees, which makes the identification of a specific ANN architecture with decision trees, allowing these to be learned by ANN methods (such as Gradient Descent backpropagation). From this we can construct Random Forests and Gradient Boosted Decision Trees from solely Neural Network topologies.

If everything can be expressed as an Artificial Neural Network, what defines an Artificial Neural Network?