Are Neural Networks Mixture Models?

Question

To my understanding, Gaussian Mixture models are a set of parameterized gaussian distributions that collectively describe an entire, aggregate distribution.

^ from McGonagle et al

Also to my understanding, in a neural network classifier with 1 hidden layer, you have a mixture of functions (sigmoids, relus, etc) that are aggregated into a function that produces a high value for things that belong to a given class (cars, planes, etc)

^ a neural network with 5 sigmoid hidden nodes

So my question is: do neural networks fall in the general domain of mixture models?

If so, why are they never referred to as such?

If not, how come?

Is it because they don't use probability distributions per se (even though a sigmoid looks a lot like the cumulative density function of a gaussian)

Just curious; thanks for any advice

Answer 1

They both fall into the general domain of graphical models.

As you've pointed out, they are very similar to each other, for they both have hidden layers and both require iterative methods to perform inference tasks.

But they are proposed on different initial ideas. "Neural network" was originally proposed by the connectionists and is now very active in the machine learning community, while "mixture model", or more general "latent variable models", is a category of classical models in the statistics community.

Neural network (in machine learning) focus mainly on minimizing the prediction error, as long as the prediction error is minimized, it doesn't matter how you interpret the mathematic equations, or how many hidden layer/nodes you used in the model. On the other hand, mixture model (in statistics) focus mainly on maximizing the marginal likelihood, and every hidden layer and node matters because each of the hidden node or layer must have a corresponding real world explanation.

The difference in initial purpose lead to some minor differences in the math equations and terms. For example the "activation function" in neural networks plays the same role as "conditional probability distribution function" in mixture models.

Nowadays there's a tendency to unify the terms in different community with graphical model language. For example from graphical model perspective, no matter it's "activation function" or "conditional probability distribution function", they are all called "factors".

Answer 2

no, neural network per-se are not mixture models, tho ideas from mixture models has influenced some of the neural network modules like softmax attention.

mixture models require a mixing in the probability density function, corresponding to the logsumexp() function. Some NN uses logsumexp, on pdf quantities and non-pdf quantities.

Thus NN are not necessarily MM, but may use MM subcomponents, and vice versa.

Answer 3

Also to my understanding, in a neural network classifier with 1 hidden layer, you have a mixture of functions (sigmoids, relus, etc) that are aggregated

...

So my question is: do neural networks fall in the general domain of mixture models?

If so, why are they never referred to as such?

You can consider a single layer models as a mixture model. And it is not true that they are never referred to as such. The link between neural networks and mixture models is not an unmentioned topic in the literature: Why is the posterior of a neural network gaussian process equal to the posterior of a neural network in the limit of infinite width layers? and for special cases, neural network Gaussian process, also deep networks are in the limit equivalent to a mixture model.

But, neural networks can be more complex and compound several layers together. Also, interactions might be non-linear.

So while there are probably historical reasons for the term neural network, neural networks are also a lot different from mixture models and in practice they are different techniques.

Neural networks are more about creating networks with multiple complex links, where mixture models are more straightforward single layer models with predefined elements to be mixed.

To take your example with the 5 layer model (which I believe can be expressed in 3 layers as well) the neural network is about blending the layers together. I have tried to re-express it in the image below (this is my own brain doing the links in the neural network):

In the last layer you have two nodes E and F that give an output between 0 and 1 depending on the values of the nodes before that C and D, but effectively they are functions of the original input X and Y and in some sense you can see it as a model that adds up a mixture of functions with extremely many parameters.

The power of network models is that

• With every layer you exponentially grow the number of elements in the mixture.
• You don't define the functions that are being mixed. Instead out you let the network define these functions out of a much larger infinite space of potential functions.