Things can be quite confusing because: 1) there are many types of models described within the neural network literature, and 2) these models overlap significantly with models in the statistics literature. I'll mainly focus here on categorizing some of the major variations using terminology from the neural networks side.
Here let's define neural networks generally as any graph/network of nodes/units/variables. This leaves a lot of room for variations. To categorize the various types, it helps me to consider a set of relatively independent binary axes:
- Link type: directed vs undirected
- Unit activation support: discrete vs real
- Unit activation function: deterministic vs probabilistic
- Overall model type: conditional vs unconditional
- Learning goal: point estimate vs distribution
Some common examples:
Certain variations are straightforward. For example, Hopfield nets, Boltzmann machines, and sigmoid belief networks can easily be treated as either conditional or unconditional models. Some combinations may be interesting even if not popular in the literature, at least as thought experiments. (Try randomly choosing one of the $2^5$ combinations, and see what happens...)
Other axes might be useful too, but I've found these cover most of the cases that interest me. Another implicit axis for any neural network is the graph structure (usually a sequence of layers for directed models, sometimes a grid for undirected models, etc.)
One example that doesn't totally fit here is self-organizing maps and other models with some smooth manifold assumption on the model structure... so another "axis" could be a choice of manifold topology. Another example that doesn't quite fit is K-nearest neighbors. Roughly, this is like a single-layer neural network (dir-real-det-uncond-point?) with an extra "lateral inhibition" mechanism to implement the winner-take-all competition. So there's an initial real-valued activation phase, but then the competition produces a discrete result.
Regarding linear vs. non-linear models: Feedforward neural networks (multilayer perceptrons) can be linear models if all units have linear activation functions, in which case a single layer is sufficient. Generally, with non-linear activation functions, they are non-linear models.
A few more thoughts:
- Training standard "neural networks" (multilayer perceptrons) = learning high-dimensional real-valued deterministic functions.
- Supervised learning = using conditional models. Unsupervised learning = using unconditional models.
- Within the "probabilistic graphical model" literature (e.g. see Koller & Friedman), Boltzmann machines are a special case of "undirected graphical model," and sigmoid belief networks are a special case of "directed graphical model" (aka Bayesian networks).
- To understand any given type of model, try to think in abstract mathematical terms about what exactly is being modeled.