# Gaussian mixture models with constrained mixing proportions

I am fitting a Gaussian mixture model to multivariate data and my application suggests constraining the mixing proportions to lie in a pre-determined sub-space. I am curious if such an approach has been tried in other application areas and if there is literature that discusses its pros and cons. I could find extensive literature on GMMs with means constrained to a pre-determined subspace but I couldn't find examples of GMMs with constrained mixing proportions. I would be grateful to know if there is a good reason why this isn't a commonly pursued modelling approach (or examples/articles of such an approach), given that the constrained parameter GMMs are common and well understood.

A note on the EM procedure that I am currently using to fit the GMM with constrained mixing proportions:

Let p = (p1, p2 ...pK) be the mixing proportions for the K clusters and $\frac{exp(P^T\mathbf{B_{i}})}{\sum_{k}exp(P^T\mathbf{B_{k}})}$ = pi define the constraint on p, where P ${\in}$ $\mathbb{R}^q$ is given, B ${\in}$ $\mathbb{R}^{qXK}$ is unknown

1) E-step: Compute responsibilities using current values of the parameters

2) M-step: Assign observations to clusters using maxkqik, the computed responsibilities for the observation i. Fit a multi response logistic regression (predictor P and cluster counts as the multinomial response) to obtain ${\hat{\mathbf{B}}}$. Set pn+1 <- $\frac{exp(P^T\mathbf{{\hat{B}}})}{\sum_{k}exp(P^T\mathbf{{\hat{ B_{k}}} })}$

• I think that there is such a general framework: It is called 'Bayesification'. Let us simplify in order to see what is going on: Let us assume that we have only K=1 cluster, i.e. we want to fit a normal distribution to some data $x_1, ..., x_N$. Suppose we know already that the mean should be 'around 0' (maybe between -1 and +1) and the second parameter $\sigma$ should be in $[0.5, 1.5]$. Then we cannot simply fit (i.e. maximize) $p(x|\theta) = \prod_{i=1}^N p(x_i|\theta) = \prod_{i=1}^N f(x_i; \mu, \sigma^2)$ where $f$ is the density of the normal distribution because then the ... – Fabian Werner Jul 18 '18 at 8:41
• optimizer will find whatever parameter best explains the data (and that might result in $\mu$ being far away from $0$ and $\sigma$ being far away from the interval $[0.5, 1.5]$). You need to 'guide' the optmizer into the right direction by putting a so-called prior on $\mu, \sigma$. One possible (well working) prior is the normalized inverse gamma distribution (see en.wikipedia.org/wiki/Conjugate_prior). Then you select parameters for this normalized inverse gamma distribution that puts much mass on the interval $[-0.5, 0.5] \times [0.5, 1.5]$. You can do the same thing with GMMs. – Fabian Werner Jul 18 '18 at 8:45
• I was thinking of the constraint as a regularizer. True, regularization has a Bayesian interpretation. Thanks. – datameddler Jul 19 '18 at 5:04

Constraining the $p_i$'s in a Gaussian mixture model with density $$\sum_{i=1}^k p_i \varphi(x;\mu_i,\sigma_i)$$ is feasible, in the sense that this simply induces a reparameterisation of the $p_i$'s in terms of another parameter $\xi$ within a set of smaller dimension than the $\mathbb{R}^k$ simplex$$p_i=p_i(\xi)\qquad i=1,...,k$$ Implementation via a Metropolis-within-Gibbs algorithm should be feasible when the transform $\mathbf{p}(.)$ is easy to compute.