1
$\begingroup$

I was wondering what is the difference between Mixture models and Mixed models in Statistics? Explaining with Any practical example would be appreciated.

$\endgroup$
  • $\begingroup$ They're totally different. Mixed models treat some terms as fixed while treating others as random (i.e.: subjecting them to shrinkage). Mixture models say that there are several different data-generating processes, and each observation is derived from a mixture of these. $\endgroup$ – generic_user Oct 26 '17 at 13:46
3
$\begingroup$

Tim gives a good answer describing the conceptual differences between the two model classes. In the interest of completeness, since you asked for a practical example, here is some R code for generating data from a mixture model. More specifically, this is a Gaussian mixture model with two components; to adapt this to Tim's notation so you can see the relationship, we have that:

$$ g(x) = \sum_{k=1}^K \pi_k f_k(x; \vartheta_k) $$

Where $K=2$ and $f_k(x; \vartheta_k) \sim N(\mu_k,\sigma^2_k)$. That is, $g(x)$ is distributed as a finite mixture of two normal (Gaussian) distributions, each with its own mean and variance; where $\pi$ is a parameter that governs the degree of mixing. Now, let's visualize what this all actually means. Let's start by setting some of these parameters to fixed values.

Let's say that $\pi=0.5$, $\mu_1=1$, $\sigma^2_1=1$, $\mu_2=6$, and $\sigma^2_2=2$. This gives us:

$$ g(x) = 0.5\times N(1,1) + 0.5\times N(6,2) $$

So you can see that this is just a weighted sum of two normal distributions. Let's see what happens when we generate this data in R:

# Set our sample size
N <- 1000

# Set our values of pi
pi  <- sample(1:2,prob=c(0.5,0.5),size=N,replace=TRUE)

# Set the parameters of our two normal distributions
mus <- c(1,6)
sds <- sqrt(c(1,2))
# Note that above I parameterized our normals in terms of their
# variance, but the rnorm function below requires standard
# deviations, thus why I'm taking the square root.

# Generate our data
mixture_model <- rnorm(n=N,mean=mus[pi],sd=sds[pi])

# Histogram
hist(mixture_model)

enter image description here You can see that we have created a distribution that is bimodal, with each mode corresponding to one of our component means (1 and 6). I will leave it as an exercise to you to see what happens as you change the mixing proportions ($\pi$) or the parameters of each component, and how that impacts the mixture distribution. It is also possible to define mixtures with more than two components (in fact, there is even a literature on "infinite" mixtures, where the number of components is in itself a random variable!) and with distributions other than normal ones (indeed, in general, there is no need even for each component of the mixture to be the same distribution, and I have even seen nested mixture models, where one component of the mixture is in and of itself another mixture model!).

(As a sidebar, the term "mixture model" has occasionally been used to describe a different class of models that are more properly referred to as "compound probability distributions". For example, if we have a Poisson distribution whose rate parameter is assumed to be a random variable following a Gamma distribution, the resulting "Poisson-Gamma mixture," as it is occasionally called, is actually a compound probability distribution that can be shown to follow a negative binomial distribution. There is a relationship here with the notion of prior/posterior distributions in Bayesian models, and with the notion of finite mixture models I described above, but that's beyond the scope of this question.)

Now, what about mixed models (i.e. mixed-effect models)? Well, as alluded to by Tim, mixed models are really regression models, where we make specific assumptions about the nature of the regression parameters (i.e. fixed vs. random effects). In general, a mixed model is any regression model that contains both fixed and random effects, where we assume the random effects follow some distribution. See the link in Tim's answer for a more thorough discussion on what this actually means.

The main conceptual difference between the approaches is that a mixture model is really just a way of specifying the distribution of a random variable (as being a mixture of other distributions), while mixed models are a way of specifying the relationship between a set of covariates and an outcome variable. Indeed, it is possible to have a mixed(-effect) mixture model, where we have the outcome variable following a mixture of distributions and we try to relate a set of covariates to that mixture.

$\endgroup$
  • $\begingroup$ @ Ryan Simmons, thanks so much for your great advice. Just another question, Is there any difference between mixed models and mixed distributions? $\endgroup$ – user178953 Oct 26 '17 at 18:50
  • $\begingroup$ @user178953 I'm not sure what you mean by "mixed distributions", since that's not really a standard term. Regardless, "mixed models" refer to the architecture of a regression model, independent of distributional assumptions. $\endgroup$ – Ryan Simmons Oct 26 '17 at 19:27
  • $\begingroup$ @ in one of the article which I am reading now, the author mentioned that "In mixed distributions, there are several Gaussian (or other) components, each with different parameters." Do you know why they used mixed distribution ( instead of mixture)? $\endgroup$ – user178953 Oct 26 '17 at 20:31
5
$\begingroup$

Besides similar sounding names, they are completely different kind of models.

Finite mixture models are models that describe your data in terms of mixture distribution,

$$ g(x) = \sum_{k=1}^K \pi_k f_k(x; \vartheta_k) $$

where the final distribution $g$ is a mixture of $K$ component-distributions $f_k$ parametrized by own parameters $\vartheta_k$ and mixing proportion $\pi_k \ge 0$ such that $\sum_{k=1}^K \pi_k = 1$. They can be used for many different pourposes like clustering, but also there are more complicated useages like cluster-wise regression. There are also infinite mixtures, where there is no fixed $K$, but this is a longer story.

Mixed effects models and generalized mixed effects models are similar to linear regression and generalized linear models, but as regression and GLM's include only fixed effects, LMM's and GLMM's include also random effects. For more details see What is the difference between fixed effect, random effect and mixed effect models?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.