# Difference between a mixture model and ensembling

Can someone explain to me the difference between a mixture model and a model ensemble? It seems both of them incorporate multiple models.

The term "mixture model" typically means a model of a dataset that is thought to contain several subpopulations. The subpopulations may have the same general distributional form but with differences in parameter values such as means or variances. The task is then to find a combination of such subpopulations that best describe the data. This page shows an example of mixture modeling in practice. See the tag for finite mixture models for references.

The term "model ensemble" is used in a situation where you are trying to find the relationship between outcome variables (a continuous variable, known class membership, occurrence/timing of events) and a set of predictor variables. The ensemble combines predictions from several models into a final prediction. Sometimes, as in most of the methods mentioned on this page, the various models are of the same general form (e.g., decision trees) and only provide weak predictions individually, but when combined they provide superior performance. In other types of ensembles even the types of individual models can be different.

As one example of the difference in application, you could use a mixture model to try to identify clusters of related cases within a population. You could use a model ensemble if you already knew the cluster membership and wanted to see how other variables were related to cluster membership.

As often happens, of course, there can be confusing terminology. For example the "mixture of experts" talked about on this page is really a model ensemble, not a mixture model as the term is generally used.

They are very different, with the most obvious difference of several being in the order of operations such that in ensemble modeling there are two or more processed models, the results of which are then combined, and, for mixture models there is one model with two or more combined terms that is then processed.

Ensemble modeling is the process of running two or more related but different analytical models and then synthesizing the results into a single score or spread in order to improve the accuracy of predictive analytics and data mining applications.

For mixture models, using the simplest example of having only two models for random variables, there is only one model consisting of the arithmetic sum of two scaled models for random variables (Do not confuse the sum of two models for random variables with the pairwise outcome sum of two random variables). For example for two normal distributions, a mixture model is

$$p\;\mathcal{N}(\mu_1,\sigma_1^2|x)+(1-p)\;\mathcal{N}(\mu_2,\sigma_2^2|x)=\\ \frac{p}{\sqrt{2 \pi } \sigma_1 }e^{-\dfrac{(x-\mu_1 )^2}{2 {\sigma_1} ^2}}+\frac{1-p}{\sqrt{2 \pi } \sigma_2 }e^{-\dfrac{(x-\mu_2 )^2}{2 {\sigma_2} ^2}}$$.

Note posted due to comments: 1) This sum is not a joint list sum, there are no paired discrete values in a list, and $$x$$ is not random, it is simple independent variable defined on $$(-\infty,\infty)$$. 2) The correct notation for the convolution of two normal distribution models is written as $$\mathcal{N}(\mu_1,\sigma_1^2|u)\ast\mathcal{N}(\mu_1,\sigma_1^2|u)(x)=\\ \frac{e^{-\dfrac{(\mu_1+\mu_2-x)^2}{2 \left(\sigma_1^2+\sigma_2^2\right)}}}{\sqrt{2 \pi } \sigma_1 \sigma_2 \sqrt{\dfrac{1}{\sigma_1^2}+\dfrac{1}{\sigma_2^2}}}$$ Note the dummy variable $$u$$, and the presence of the convolution operator $$\ast$$ rather than a '+' sign. Some people use $$\star$$ with or without the required dummy variable. In any case, it is not a sum, and convolutions are only sums for joint random variable outcomes undergoing addition, one outcome pair at a time, such that unlike a mixture RV which can have $$n\neq m$$ and $$n+m$$ outcomes, a finite sum convolution of two RV's has $$2n$$ entries and $$n$$ sums. To turn a continuous process convolution written as an integral into a summation with uncountable many additions is possible in the Riemann sense, but that is not the sense here. 3) A normal distribution does not have to be used as a model for a random variable at all. It might, for example, be used as a model for bacterial density versus linear displacement from the center of bacterial colony in a Petri dish, and bacteria per square mm are just not probabilities.

There appears to be considerable confusion between what density functions are and what random variables are. Note 1) a density function is not always a probability density function and whatever notation is used must allow for both contingencies. 2) Density functions of the continuous type are often used as models for physical situations and in different circumstances are often used as models for random variables. There are many who, at the present time, refer to density functions as if they were all random variables. It is most of the time misplaced terminology in the sense that more frequently a pdf of a random variable, is a model for that random variable, and is not itself random in any sense.

The take home on this, as told to me by my Easter bunny, is that "If a random variable were an egg, a pdf would be a Fabergé egg."

• (-1) In the 2nd paragraph you've confused summing random variables (to give a new random variable) with summing their weighted density functions (to give a new density function, that of the mixture distribution), though the explanation in parentheses is all right. The mathematical expression you've provided is for the density function, given that $\mathcal{N}$ is the normal density function. (Also, I don't know what you mean by "unpaired random variables" - "independent random variables" perhaps?) – Scortchi Apr 7 at 10:12
• Sorry, it's the third paragraph where you refer to the "sum of two unpaired random variables" – Scortchi Apr 7 at 19:52
• Well, to take up your example, the sum of two random variables having normal distributions is another random variable having a normal distribution, not a mixture of two normal distributions. – Scortchi Apr 7 at 20:10
• Provided they're independent (which I should've mentioned), they do. See Sum of normally distributed random variables & note the 2nd paragraph: "This is not to be confused with the sum of normal distributions which forms a mixture distribution". – Scortchi Apr 8 at 12:24
• @Scortchi The sum of two scaled density function models for random variables is not random, it is just a function. I have corrected my wording to be more exact and explained the difference. – Carl Apr 22 at 2:02