1
$\begingroup$

I am fitting a Gaussian mixture model to multivariate data and my application suggests constraining the mixing proportions to lie in a pre-determined sub-space. I am curious if such an approach has been tried in other application areas and if there is literature that discusses its pros and cons. I could find extensive literature on GMMs with means constrained to a pre-determined subspace but I couldn't find examples of GMMs with constrained mixing proportions. I would be grateful to know if there is a good reason why this isn't a commonly pursued modelling approach (or examples/articles of such an approach), given that the constrained parameter GMMs are common and well understood.

A note on the EM procedure that I am currently using to fit the GMM with constrained mixing proportions:

Let p = (p1, p2 ...pK) be the mixing proportions for the K clusters and $\frac{exp(P^T\mathbf{B_{i}})}{\sum_{k}exp(P^T\mathbf{B_{k}})}$ = pi define the constraint on p, where P ${\in}$ $\mathbb{R}^q$ is given, B ${\in}$ $\mathbb{R}^{qXK}$ is unknown

1) E-step: Compute responsibilities using current values of the parameters

2) M-step: Assign observations to clusters using maxkqik, the computed responsibilities for the observation i. Fit a multi response logistic regression (predictor P and cluster counts as the multinomial response) to obtain ${\hat{\mathbf{B}}}$. Set pn+1 <- $\frac{exp(P^T\mathbf{{\hat{B}}})}{\sum_{k}exp(P^T\mathbf{{\hat{ B_{k}}} })}$

$\endgroup$
  • $\begingroup$ I think that there is such a general framework: It is called 'Bayesification'. Let us simplify in order to see what is going on: Let us assume that we have only K=1 cluster, i.e. we want to fit a normal distribution to some data $x_1, ..., x_N$. Suppose we know already that the mean should be 'around 0' (maybe between -1 and +1) and the second parameter $\sigma$ should be in $[0.5, 1.5]$. Then we cannot simply fit (i.e. maximize) $p(x|\theta) = \prod_{i=1}^N p(x_i|\theta) = \prod_{i=1}^N f(x_i; \mu, \sigma^2)$ where $f$ is the density of the normal distribution because then the ... $\endgroup$ – Fabian Werner Jul 18 '18 at 8:41
  • $\begingroup$ optimizer will find whatever parameter best explains the data (and that might result in $\mu$ being far away from $0$ and $\sigma$ being far away from the interval $[0.5, 1.5]$). You need to 'guide' the optmizer into the right direction by putting a so-called prior on $\mu, \sigma$. One possible (well working) prior is the normalized inverse gamma distribution (see en.wikipedia.org/wiki/Conjugate_prior). Then you select parameters for this normalized inverse gamma distribution that puts much mass on the interval $[-0.5, 0.5] \times [0.5, 1.5]$. You can do the same thing with GMMs. $\endgroup$ – Fabian Werner Jul 18 '18 at 8:45
  • $\begingroup$ I was thinking of the constraint as a regularizer. True, regularization has a Bayesian interpretation. Thanks. $\endgroup$ – datameddler Jul 19 '18 at 5:04
1
$\begingroup$

Constraining the $p_i$'s in a Gaussian mixture model with density $$\sum_{i=1}^k p_i \varphi(x;\mu_i,\sigma_i)$$ is feasible, in the sense that this simply induces a reparameterisation of the $p_i$'s in terms of another parameter $\xi$ within a set of smaller dimension than the $\mathbb{R}^k$ simplex$$p_i=p_i(\xi)\qquad i=1,...,k$$ Implementation via a Metropolis-within-Gibbs algorithm should be feasible when the transform $\mathbf{p}(.)$ is easy to compute.

$\endgroup$
  • $\begingroup$ I have tried fitting using EM with a modification. $\endgroup$ – datameddler Jul 19 '18 at 4:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.