Estimate Gaussian (mixture) density from a set of weighted samples

I am trying to find if there exists a way to find the spatial distribution over a set of points where the points are weighted. If I have "n" points in the (x,y) space, I can fit a mixture of Gaussians to estimate a distribution. However, there might be situations where some points are more important than others so I have weights associated with my points. For example, the k-means algorithm can be extended to a weighted k-means problem. Is there a way to do so with a mixture of Gaussians?

SO has a post on this here but there are no accepted answers. I am having a similar problem and I thought this is a better forum.

• Before to write up a long answer - if you look closely into how EM works you will see that in each iteration $\vec{x}$ is used to compute the means per group. You can use the $\vec{x}\cdot\vec{w}_x$ there. Alternatively (or even additinally... need to think about it more) you need to include these terms into the computation of the log-likelihood. But given that... you can scale your samples according to the weights before applying EM and then run it. The result will be the same (because log-likelihood is used as objective function) – Drey Nov 25 '16 at 13:12

In case there is no "standard" weighted GMM in the literature (I'm suprised I haven't found anything), I would try the following:

Let:

• $z_i$ be the cluster of point $i$
• $w_i$ be the observed weight $[0, \inf)$
• $\boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k$ the mean and covariance of the cluster $k$

and knowing that a non-weighted GMM considers that points in are generated according to:

\begin{align} \mathbf{x_i} \sim \mathcal{N}(\boldsymbol{\mu}_{z_i}, \boldsymbol{\Sigma}_{z_i})\\ \end{align}

you could modify this generative process a little bit and say that points with less weight have more freedom to choose their location. That is, you give them more variance. It might be:

\begin{align} \mathbf{x_i} \sim \mathcal{N}(\boldsymbol{\mu}_{z_i}, e^{w_i}\boldsymbol{\Sigma}_{z_i})\\ \end{align}

or

\begin{align} \mathbf{x_i} \sim \mathcal{N}(\boldsymbol{\mu}_{z_i}, (1+w_i)\boldsymbol{\Sigma}_{z_i})\\ \end{align}

• Thanks. I do not quite get the idea behind having freedom to choose locations. The points already have fixed locations, don't they? – statBeginner Mar 8 '15 at 18:42
• Does Jakob's answer from the SO post make sense to you? Here is what we wrote : If $v_i$ is the weight of the i-th sample, the algorithm from the tutorial (see end of Section 6.2.) changes so that the $\gamma_{ij}$ is multiplied by that weighting factor. For the calculation of the new weights $w_j$, $n_j$ has to be divided by the sum of the weights $\sum_{i=1}^{n} v_i$ instead of just n. – statBeginner Mar 8 '15 at 18:43
• If $\gamma$ is the probability of belonging to a cluster I don't see how multiplying by a weighting factor solves the problem. (Maybe, but it doesn't make sense to me, what is the interpretation?) – alberto Mar 8 '15 at 18:54
• About the freedom. The trick is that your prior on $x_i$ is more flexible, less dogmatic, if $x_i$ has a small weight, so the algorithm wouldn't care as much about these points. – alberto Mar 8 '15 at 18:56
• Thanks, I see what you are trying to say now. About multiplying the weights, i think what he is trying to say is that after you calculate the probabilities, you rescale them by multiplying them with the weights. Then to normalize, instead of dividing by $n$, you divide by the sum of weights. – statBeginner Mar 8 '15 at 19:25