To my understanding, Gaussian Mixture models are a set of parameterized gaussian distributions that collectively describe an entire, aggregate distribution.
gaussian mixture gif
^ from McGonagle et al

Also to my understanding, in a neural network classifier with 1 hidden layer, you have a mixture of functions (sigmoids, relus, etc) that are aggregated into a function that produces a high value for things that belong to a given class (cars, planes, etc)
neural network gif
^ a neural network with 5 sigmoid hidden nodes

So my question is: do neural networks fall in the general domain of mixture models?

If so, why are they never referred to as such?

If not, how come?

Is it because they don't use probability distributions per se (even though a sigmoid looks a lot like the cumulative density function of a gaussian)

Just curious; thanks for any advice

  • $\begingroup$ in your first animation with the yellow and red clusters, what is GMM doing there exactly? is it improving its fit on the mixture components as its parameters are optimized? or is GMM actively honing in on the groups somehow? I've only seen static GMM charts $\endgroup$
    – develarist
    Jul 9, 2020 at 12:53
  • $\begingroup$ @develarist I think the animation is showing a GMM being optimized with the Expectation Maximization algorithm (en.m.wikipedia.org/wiki/Expectation–maximization_algorithm), which iteratively updates the clusters according to the optimization objective. $\endgroup$
    – Matt
    Jul 13, 2020 at 20:36
  • $\begingroup$ Is the x axis showing the values of the mean of duration, and y axis the values of the mean of delay $\endgroup$
    – develarist
    Jul 13, 2020 at 23:53
  • $\begingroup$ @develarist yes; I think so. duration and delay are just two features in the dataset being visualized. though I would note, in practice, GMMs don't always converge to the 'global' means; rather, they move to local regions that are densely packed with data points. Sometimes, those are the 'global' means; other times, they aren't. $\endgroup$
    – Matt
    Jul 15, 2020 at 14:02

3 Answers 3


They both fall into the general domain of graphical models.

As you've pointed out, they are very similar to each other, for they both have hidden layers and both require iterative methods to perform inference tasks.

But they are proposed on different initial ideas. "Neural network" was originally proposed by the connectionists and is now very active in the machine learning community, while "mixture model", or more general "latent variable models", is a category of classical models in the statistics community.

Neural network (in machine learning) focus mainly on minimizing the prediction error, as long as the prediction error is minimized, it doesn't matter how you interpret the mathematic equations, or how many hidden layer/nodes you used in the model. On the other hand, mixture model (in statistics) focus mainly on maximizing the marginal likelihood, and every hidden layer and node matters because each of the hidden node or layer must have a corresponding real world explanation.

The difference in initial purpose lead to some minor differences in the math equations and terms. For example the "activation function" in neural networks plays the same role as "conditional probability distribution function" in mixture models.

Nowadays there's a tendency to unify the terms in different community with graphical model language. For example from graphical model perspective, no matter it's "activation function" or "conditional probability distribution function", they are all called "factors".

  • 1
    $\begingroup$ Thanks for the response! By chance, do you know of any papers / resources that explore the connection between neural nets, mixture models, and graphical models in depth? $\endgroup$
    – Matt
    May 5, 2020 at 12:15
  • $\begingroup$ +1 second this :) $\endgroup$ Nov 11, 2022 at 10:03

no, neural network per-se are not mixture models, tho ideas from mixture models has influenced some of the neural network modules like softmax attention.

mixture models require a mixing in the probability density function, corresponding to the logsumexp() function. Some NN uses logsumexp, on pdf quantities and non-pdf quantities.

Thus NN are not necessarily MM, but may use MM subcomponents, and vice versa.


Also to my understanding, in a neural network classifier with 1 hidden layer, you have a mixture of functions (sigmoids, relus, etc) that are aggregated


So my question is: do neural networks fall in the general domain of mixture models?

If so, why are they never referred to as such?

You can consider a single layer models as a mixture model. And it is not true that they are never referred to as such. The link between neural networks and mixture models is not an unmentioned topic in the literature: Why is the posterior of a neural network gaussian process equal to the posterior of a neural network in the limit of infinite width layers? and for special cases, neural network Gaussian process, also deep networks are in the limit equivalent to a mixture model.

But, neural networks can be more complex and compound several layers together. Also, interactions might be non-linear.

So while there are probably historical reasons for the term neural network, neural networks are also a lot different from mixture models and in practice they are different techniques.

Neural networks are more about creating networks with multiple complex links, where mixture models are more straightforward single layer models with predefined elements to be mixed.

To take your example with the 5 layer model (which I believe can be expressed in 3 layers as well) the neural network is about blending the layers together. I have tried to re-express it in the image below (this is my own brain doing the links in the neural network):

example of the mixing

In the last layer you have two nodes E and F that give an output between 0 and 1 depending on the values of the nodes before that C and D, but effectively they are functions of the original input X and Y and in some sense you can see it as a model that adds up a mixture of functions with extremely many parameters.

The power of network models is that

  • With every layer you exponentially grow the number of elements in the mixture.
  • You don't define the functions that are being mixed. Instead out you let the network define these functions out of a much larger infinite space of potential functions.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.