Where do artificial neural networks belong in the 'taxonomy' of statistical learning methods?

Question

I'm a non-stats person trying to learn more about statistical learning methods, and to organize my thinking I am trying to construct a mental taxonomy of the methods I'm learning about. For instance:

Statistical learning methods can be divided into supervised and unsupervised categories.

Supervised methods can be divided into linear and non-linear methods. Supervised linear methods include generalized linear models and their derivations. Non-linear methods include tree-based methods, support vector machines, etc.

Unsupervised methods include K-nearest neighbours, PCA, etc.

What I'm struggling with is the position of artificial neural networks in this taxonomy.

Neural networks allow us to reframe the variation in a dataset -- so are they an unsupervised learning method, similar to PCA? However, we can train them to perform a classification task -- so does that make them a supervised method? Or, are they better understood as an implementation of statistical learning methods? For instance, I have seen (but not necessarily understood) references to using support vector machines within a neural network.

My apologies if this is a basic question. I have tried reading around it and remain confused but direction to further resources would be appreciated. I'm working through ISLR, and in part my confusion stems from the fact that that text makes no mention of neural networks.

Answer 1

Neural nets form a broad class of models, and cover many parts of the taxonomy you're describing (and even extend outside it). An individual neural net (or subclass of them) could be placed into the taxonomy, but the entire class of neural nets cannot.

For example, neural nets can be used for both supervised and unsupervised learning problems, depending on the loss function. They can be linear or nonlinear, depending on the network architecture and activation function. Many other models (e.g. linear/logistic regression, kernel machines, PCA, etc.) are equivalent to a particular form of neural net.

Neural nets can also be used to solve problems that are not statistical learning problems at all. For example, Hopfield nets can be used to solve optimization problems. There are even computationally universal neural nets that can implement every possible algorithm (see here, but this is a theoretical construction that would not be used in practice).

Answer 2

Things can be quite confusing because: 1) there are many types of models described within the neural network literature, and 2) these models overlap significantly with models in the statistics literature. I'll mainly focus here on categorizing some of the major variations using terminology from the neural networks side.

Here let's define neural networks generally as any graph/network of nodes/units/variables. This leaves a lot of room for variations. To categorize the various types, it helps me to consider a set of relatively independent binary axes:

• Link type: directed vs undirected
• Unit activation support: discrete vs real
• Unit activation function: deterministic vs probabilistic
• Overall model type: conditional vs unconditional
• Learning goal: point estimate vs distribution

Some common examples:

• Feedforward neural networks (aka multilayer perceptrons) optimization: dir-real-det-cond-point
• Hopfield networks: undir-disc-det-uncond-point
• (Unconditional) Boltzmann machines: undir-disc-prob-uncond-point
• (Conditional) Sigmoid belief networks: dir-disc-prob-cond-point
• PCA: undir-real-det-uncond-point?
• ...

Certain variations are straightforward. For example, Hopfield nets, Boltzmann machines, and sigmoid belief networks can easily be treated as either conditional or unconditional models. Some combinations may be interesting even if not popular in the literature, at least as thought experiments. (Try randomly choosing one of the $2^5$ combinations, and see what happens...)

Other axes might be useful too, but I've found these cover most of the cases that interest me. Another implicit axis for any neural network is the graph structure (usually a sequence of layers for directed models, sometimes a grid for undirected models, etc.)

One example that doesn't totally fit here is self-organizing maps and other models with some smooth manifold assumption on the model structure... so another "axis" could be a choice of manifold topology. Another example that doesn't quite fit is K-nearest neighbors. Roughly, this is like a single-layer neural network (dir-real-det-uncond-point?) with an extra "lateral inhibition" mechanism to implement the winner-take-all competition. So there's an initial real-valued activation phase, but then the competition produces a discrete result.

Regarding linear vs. non-linear models: Feedforward neural networks (multilayer perceptrons) can be linear models if all units have linear activation functions, in which case a single layer is sufficient. Generally, with non-linear activation functions, they are non-linear models.

A few more thoughts:

• Training standard "neural networks" (multilayer perceptrons) = learning high-dimensional real-valued deterministic functions.
• Supervised learning = using conditional models. Unsupervised learning = using unconditional models.
• Within the "probabilistic graphical model" literature (e.g. see Koller & Friedman), Boltzmann machines are a special case of "undirected graphical model," and sigmoid belief networks are a special case of "directed graphical model" (aka Bayesian networks).
• To understand any given type of model, try to think in abstract mathematical terms about what exactly is being modeled.