To my understanding, Gaussian Mixture models are a set of parameterized gaussian distributions that collectively describe an entire, aggregate distribution.
gaussian mixture gif
^ from McGonagle et al

Also to my understanding, in a neural network classifier with 1 hidden layer, you have a mixture of functions (sigmoids, relus, etc) that are aggregated into a function that produces a high value for things that belong to a given class (cars, planes, etc)
neural network gif
^ a neural network with 5 sigmoid hidden nodes

So my question is: do neural networks fall in the general domain of mixture models?

If so, why are they never referred to as such?

If not, how come?

Is it because they don't use probability distributions per se (even though a sigmoid looks a lot like the cumulative density function of a gaussian)

Just curious; thanks for any advice

  • $\begingroup$ in your first animation with the yellow and red clusters, what is GMM doing there exactly? is it improving its fit on the mixture components as its parameters are optimized? or is GMM actively honing in on the groups somehow? I've only seen static GMM charts $\endgroup$ – develarist Jul 9 '20 at 12:53
  • $\begingroup$ @develarist I think the animation is showing a GMM being optimized with the Expectation Maximization algorithm (en.m.wikipedia.org/wiki/Expectation–maximization_algorithm), which iteratively updates the clusters according to the optimization objective. $\endgroup$ – Matt W Jul 13 '20 at 20:36
  • $\begingroup$ Is the x axis showing the values of the mean of duration, and y axis the values of the mean of delay $\endgroup$ – develarist Jul 13 '20 at 23:53
  • $\begingroup$ @develarist yes; I think so. duration and delay are just two features in the dataset being visualized. though I would note, in practice, GMMs don't always converge to the 'global' means; rather, they move to local regions that are densely packed with data points. Sometimes, those are the 'global' means; other times, they aren't. $\endgroup$ – Matt W Jul 15 '20 at 14:02

They both fall into the general domain of graphical models.

As you've pointed out, they are very similar to each other, for they both have hidden layers and both require iterative methods to perform inference tasks.

But they are proposed on different initial ideas. "Neural network" was originally proposed by the connectionists and is now very active in the machine learning community, while "mixture model", or more general "latent variable models", is a category of classical models in the statistics community.

Neural network (in machine learning) focus mainly on minimizing the prediction error, as long as the prediction error is minimized, it doesn't matter how you interpret the mathematic equations, or how many hidden layer/nodes you used in the model. On the other hand, mixture model (in statistics) focus mainly on maximizing the marginal likelihood, and every hidden layer and node matters because each of the hidden node or layer must have a corresponding real world explanation.

The difference in initial purpose lead to some minor differences in the math equations and terms. For example the "activation function" in neural networks plays the same role as "conditional probability distribution function" in mixture models.

Nowadays there's a tendency to unify the terms in different community with graphical model language. For example from graphical model perspective, no matter it's "activation function" or "conditional probability distribution function", they are all called "factors".

  • $\begingroup$ Thanks for the response! By chance, do you know of any papers / resources that explore the connection between neural nets, mixture models, and graphical models in depth? $\endgroup$ – Matt W May 5 '20 at 12:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.