What *is* an Artificial Neural Network?

As we delve into Neural Networks literature, we get to identify other methods with neuromorphic topologies ("Neural-Network"-like architectures). And I'm not talking about the Universal Approximation Theorem. Examples are given below.

Then, it makes me wonder: what is the definition of an artificial Neural Network? Its topology appears to cover everything.

Examples:

One of the first identifications we make is between PCA and a linear Autoencoder with tied-weights in the encoder and decoder and thresholded activations in the bottleneck layer.

Also, a common identification is done between linear models (logistic regression in special) and a Neural Network with no hidden layer and a single output layer. This identification opens several doors.

Fourier and Taylor series? ANNs. SVM? ANN. Gaussian Process? ANN (with single hidden layer with infinite hidden units).

And so, just as easily, we can incorporate arbitrary regularized versions with specialized loss functions of these algorithms into a Neural Network framework.

But the more we dig, the more similarities appear. I just stumbled into Deep Neural Decision Trees, which makes the identification of a specific ANN architecture with decision trees, allowing these to be learned by ANN methods (such as Gradient Descent backpropagation). From this we can construct Random Forests and Gradient Boosted Decision Trees from solely Neural Network topologies.

If everything can be expressed as an Artificial Neural Network, what defines an Artificial Neural Network?

• That paper on Deep Neural Decision Trees is pretty far out there. Normally activation functions are real-valued functions, not outer products. So they're not really discussing ANNs as we normally think of them, but a mathematical generalization that is not widely used or accepted. To show an ANN is different than a decision tree, I would simply point out that all ANNs are parametric (have a finite parameter space) while trees are non-parametric (have a potentially infinite parameter space.) – olooney Aug 16 '18 at 1:10
• @olooney the Kronecker product is not an activation function, it's simply an operation on the outputs of the previous layer (like a convolution, or any other operation we define over activations). The DNDT can represent any decision tree, AND every DNDT can be represented by a decision tree. – Firebug Aug 16 '18 at 1:28
• @olooney going by your definition of activation function, Softmax isn't an activation function. – Firebug Aug 16 '18 at 3:18
• I'm not entirely sure I understand the motivation for this question. A possible, loose definition of an ANN is that it's a directed graphical model, that uses neurons (i.e., activation functions) to process inputs/outputs, and most of the time you use gradient descent to train it. When you say that "everything can be expressed as an ANN", are you specifically asking whether there is an exact mapping between the mentioned other models and ANNs? The problem is that you'll have to come up with highly modified training routines to match the optimizations. – Alex R. Aug 16 '18 at 23:26
• @Sycorax I also do, both he and Hinton hinted at it. I want to give an opportunity to answerers in the other camp to provide credible sources :) – Firebug Aug 18 '18 at 13:56

Jürgen Schmidhuber, "Deep Learning in Neural Networks: An Overview" traces the history of key concepts in neural networks and deep learning. In his view, neural networks would appear to encompass essentially any model which can be characterized as a directed graph where each node represents some computational unit. Schmidhuber is a prominent neural networks researcher, and wrote the original paper on LSTM networks with Sepp Hochreiter.

Which modifiable components of a learning system are responsible for its success or failure? What changes to them improve performance? This has been called the fundamental credit assignment problem (Minsky, 1963). There are general credit assignment methods for universal problem solvers that are time-optimal in various theoretical senses (Sec. 6.8). The present survey, however, will focus on the narrower, but now commercially important, subfield of Deep Learning (DL) in Artificial Neural Networks (NNs).

A standard neural network (NN) consists of many simple, connected processors called neurons, each producing a sequence of real-valued activations. Input neurons get activated through sensors perceiving the environment, other neurons get activated through weighted connections from previously active neurons (details in Sec. 2). Some neurons may influence the environment by triggering actions. Learning or credit assignment is about finding weights that make the NN exhibit desired behavior, such as driving a car. Depending on the problem and how the neurons are connected, such behavior may require long causal chains of computational stages (Sec. 3), where each stage transforms (often in a non-linear way) the aggregate activation of the network. Deep Learning is about accurately assigning credit across many such stages.

Shallow NN-like models with few such stages have been around for many decades if not centuries (Sec. 5.1). Models with several successive nonlinear layers of neurons date back at least to the 1960s (Sec. 5.3) and 1970s (Sec. 5.5). An efficient gradient descent method for teacher-based Supervised Learning (SL) in discrete, differentiable networks of arbitrary depth called backpropagation (BP) was developed in the 1960s and 1970s, and applied to NNs in 1981 (Sec. 5.5). BP-based training of deep NNs with many layers, however, had been found to be difficult in practice by the late 1980s (Sec. 5.6), and had become an explicit research subject by the early 1990s (Sec. 5.9). DL became practically feasible to some extent through the help of Unsupervised Learning (UL), e.g., Sec. 5.10 (1991), Sec. 5.15 (2006). The 1990s and 2000s also saw many improvements of purely supervised DL (Sec. 5). In the new millennium, deep NNs have finally attracted wide-spread attention, mainly by outperforming alternative machine learning methods such as kernel machines (Vapnik, 1995; Scholkopf et al., 1998) in numerous important applications. In fact, since 2009, supervised deep NNs have won many official international pattern recognition competitions (e.g., Sec. 5.17, 5.19, 5.21, 5.22), achieving the first superhuman visual pattern recognition results in limited domains (Sec. 5.19, 2011). Deep NNs also have become relevant for the more general field of Reinforcement Learning (RL) where there is no supervising teacher (Sec. 6).

On the other hand, I'm not sure that it's necessarily profitable to try and construct a taxonomy of mutually-exclusive buckets for machine learning strategies. I think we can say that there are perspectives from which models can be viewed as neural networks. I don't think that perspective is necessarily the best or useful in all contexts. For example, I'm still planning to refer to random forests and gradient boosted trees as "tree ensembles" instead of abstracting away their distinctions and calling them "neural network trees". Moreover, Schmidhuber distinguishes NNs from kernel machines -- even though kernel machines have some connections to NNs -- when he writes "In the new millennium, deep NNs have finally attracted wide-spread attention, mainly by outperforming alternative machine learning methods such as kernel machines ... in numerous important applications. "

• So basically every model and heuristic known in Machine Learning and Statistics today would be considered an ANN by Schmidhuber, with the distinctive nomenclature being simply given by the optimization strategy (including models with no optimization here)? – Firebug Aug 16 '18 at 0:55
• I get that, from a practical point of view, but it doesn't change the fact that pretty much every model is, strictly speaking, an ANN (I can't think of any single model that isn't). – Firebug Aug 16 '18 at 1:30
• @Firebug How would you reformat regression or (simple k-means and others) clustering problems, that are trained or placed in a 'learning environment', such that they are equal to this definition of ANN? – Martijn Weterings Aug 16 '18 at 14:36
• @Firebug I don't see how the fact that PCA can be shown to be equivalent to a particular autoencoder makes PCA "a neural network". In standard PCA we are not even using gradient descent. – amoeba Aug 16 '18 at 15:02
• @Firebug If you define "NN" as "connected computational nodes" then I guess any computation whatsoever is a NN. Not sure that's of any use but okay. – amoeba Aug 16 '18 at 19:25

If you want a basic definition of an ANN, you might say that it's a directed-graphical-model, where inputs and outputs are processed at each node via an activation function, and most of the time gradient descent is used to train it. So the question really becomes: what models out there can be expressed as graphical models?

I'm not an expert but, I believe theoretically some ANNs can be shown to be Turing complete, which means that they should be able to do any possible set of calculations (with a possible infinite number of resources, mind you).

I'm also going to interpret your question in the following way:

For any given model, can I slap together an ANN model to emulate that model, as close as possible, and in a reasonable amount of time?

A vanilla neural network can emulate a decision tree, by using heaviside step-activations. The problem is that such unit activations have zero gradient, so normal gradient descent won't work. You might say, "no problem, just use a modified form of gradient descent." However, that's still not enough. For a better example, take something like XGBOOST, which isn't just gradient-boosted forests. There's a whole lot of extra work that goes into choosing split points, pruning, optimizing for speed, etc. Maybe after enough modifications you can make a similar-looking ANN, but it's not at all clear that such an ANN would perform at least as well, nor if it's optimized to do the job.

I think that's an important point, because while it might be theoretically satisfying to conclude that ANNs can do anything, practically this might be completely useless. For example, you could try making an ANN using ReLu activations to approximate $f(x)=e^{x}$, but that's just plain dumb, as considerably more efficient and accurate methods are at your disposal.

• Thanks for the answer! Regarding the question - "For any given model, can I slap together an ANN model to emulate that model, as close as possible, and in a reasonable amount of time?"- I'm afraid to say that's not the point. The point is, ANN topology is so general it seems to cover everything, and optimization strategy doesn't seems to be able to determine what is and what isn't an ANN. Therefore the question, what defines an ANN? Because otherwise everything is, in a way, an ANN expressed in other terms. – Firebug Aug 16 '18 at 23:52
• "A vanilla neural network can emulate a decision tree, by using heaviside step-activations. The problem is that such unit activations have zero gradient, so normal gradient descent won't work. You might say, "no problem, just use a modified form of gradient descent." However, that's still not enough. [...]" - As we could assert, optimization isn't a determinant factor to the definition of what constitutes an ANN. If you can write every decision tree as a neural network (and we can do that), then we can safely say DTs are (a type of) NN, while the converse is not true. – Firebug Aug 16 '18 at 23:54
• "If you want a basic definition of an ANN, you might say that it's a directed-graphical-model, where inputs and outputs are processed at each node via an activation function, and most of the time gradient descent is used to train it. So the question really becomes: what models out there can be expressed as graphical models?" - I agree with this. Then, "Neural Network" can be interpret as the most general class of models, perhaps only less general than "Graph models", which is a superset of both Undirected and Directed Graph Models. Perhaps you could elaborate more on this ;) – Firebug Aug 16 '18 at 23:56

Perhaps, a more accurate name for ANNs is "differentiable networks", i.e. complex parametrized functions that can be optimized using gradient descent or its variant. This is a very general definition that emphasizes differentiability, but doesn't tell anything about principal ideas, tasks that it's suited for, underlying mathematical framework, etc.

Note that differentiability is a trait, not necessary the main. For example, SVM can be trained using gradient descent and thus exhibits properties of a neural/differentiable network, but the main idea is in data separation using hyperplanes. Variational autoencoder uses MLPs for encoder and decoder, but the function you optimize comes from Bayesian statistics, and so on.

There's also a few models that are often referred to as neural networks but don't use GD for learning. A good example is RBM. My guess is that the label "neural network" was attached to it mostly for historical reasons - eventually, RBM's creator is Geoffrey Hinton, and Hinton is a neural network guy, right? However, if you analyze the model you'll see that RBM's structure is a Markov net, energy-based cost function comes from statistical physics of the beginning of 20th centenary and MCMC/Gibbs sampling have been developing in parallel and totally independently from neural networks.

• Gradient-based learning has certainly played a role in the success of ANNs. But I don't see differentiability as being essential to the definition, because some ANNs are not differentiable. For example, the very first ANN (McCulloch-Pitts model) used binary threshold units. A current research topic is how to perform learning in non-differentiable ANNs like spiking nets. Or, suppose we start with a typical, differentiable ANN, but then declare that we want to minimize a non-differentiable loss function. Is it no longer an ANN? – user20160 Aug 16 '18 at 23:30
• That's exactly why I proposed an alternative definition that covers feed-foward, recurrent, recursive, convolutional networks, autoencoders, VAEs, GANs, attention and many other models that we normally call "neural networks", but excludes e.g. approaches based on simulating human brain or extensive sampling over PGMs. As of 2018, these approaches are really different, they use different optimization methods, different libraries, etc. (Although I can't think of a better name than "neural network" for spiking nets since, unlike CNNs or RNNs, actually simulate human brain). – ffriend Aug 17 '18 at 10:17