# How to calculate likelihood for a mixture model with missing data?

Toy explanation: I have set of different cars of different colours. There can be green, blue, red, etc. cars. I have a set of classes i.e.: "The set contains blue, red and pink cars" or "The set contains only green cars". Let's assume that colour of the cars from the set can be modelled as the mix of normals with known means/variances.

I can also look at the people that are sitting in the cars. For example, there can be 10 people that wear green in the green car. It increases the evidence that this car is green. Or I can have a car that has a colour between blue and purple and 3 persons who are wearing purple inside the car. I would say that the car is purple. And I can have a car without any people inside.

I would use EM algorithm if I would deal only with the colours of cars. But how to take people that can probably sit inside the car into account? The number of people inside different cars is different so the likelihoods will be different for each point (I mean that we can not just multiply likelihoods no more, some likelihoods will be extra small just because a lot of people are in the car).

How to fit Gaussian mixture on colours taking additional data points into account? There is no problem for classification of each separate point: the number of people is constant so the likelihoods will be "comparable" between different models, i.e. "This car is green" vs. "This car is blue". How to calculate the joint likelihood of all set of cars?

In other words, I have an ordered vector of values:

$(x_{1}, x_{2}, ..., x_{n})$ and each point were generated from Gaussian with fixed mean and $\sigma$. We know the possible set of means (so we can use constrained EM algorithm).

Example of model for vector of $x$s: mixture of Gaussian components with means (0, 1.0, 2.0, ..., 5.0) and $\sigma=0.05$. Another model can be mixture of components with means (2.0, 3.0, 4.0) and the same $\sigma$. My goal is to decide which model is true.

Each component with different mean can produce additional data: $x_i \leftrightarrow \{y_1, ...\}$. So if the point $x_i$ belongs to the Gaussian mixture component with mean 2.0, it can produce 0, 1 or more additional points that "says" that the pair $x_i \leftrightarrow \{y_1, ...\}$ belong to mixture component with the mean 2.0 (the likelihood $L(x,y|\mu=2.0)$ is bigger than $L(x,y|\mu \neq 2.0)$). The problem is that the number of additional points for each $x_i$ is different (it is possible to be $x_i \leftrightarrow \{\emptyset\}$). So if we will try calculate joint likelihood of $x_i \leftrightarrow \{y_1, ...\}$ just by multiplying probabilities $p(x_i|\mu=2.0)\cdot p(y_1|\mu=2.0)\cdot \ldots \cdot p(y_m|\mu=2.0)$ we will have the values of different order for different points!

UPD: Since membership weights of any point in mixture modelling are normalised to $\sum \pi_k = 1$, can I use the additional data to estimate at least mixture proportions more accurate? It seems that they will remain the same for the case when there is no additional data and it will be only better taking additional data into account. Now I do not see a way how to take into account additional data for the final likelihood calculation, but membership weights can be adjusted, if I am not wrong.

UPD1: The best idea so far: to create a sample of equal length from cars/people using bootstrap. So $x_i \leftrightarrow \{y_1, ...y_m\}$ for all $i$. If the set of $y$ is empty, we can repeat $x_i$ $m$ times. Looks super bad, but...

• Could you go into more details on how does your data look like? My initial guess is that you should use E-M algorithm somehow, but it's hard to say without understanding your problem better. – Tim Sep 21 '16 at 10:49
• @Tim thanks for the comment, I updated the question. I am trying to use EM, but I am stuck with the likelihood calculation. Each one points $x_i$ from one dimension can have an array of arbitrary size of additional data points, and for me it is not possible to calculate the joint likelihood for different number of points at each coordinate. – German Demidov Sep 21 '16 at 11:14

A possible completion for your model (as far as I understand it without proper mathematical notations) is the hierarchical structure

1. Generate index $\iota$ taking value $i$ with probability $\pi_i$
2. Generate positive integer $m$ from a fixed distribution, e.g., a shifted Poisson $1+\mathcal{P}(1)$
3. Generate $m$ iid values from $\text{N}(\mu_\iota,\sigma^2)$

If this model is acceptable, it is straightforward to write the EM algorithm for this extension of a standard mixture model. (Note that I picked a shifted Poisson $1+\mathcal{P}(1)$ as an arbitrary choice since it does not matter for inference.)

• I really appreciate your answer, but I can not understand it. What should I do next with these $m$ generated iid values? I have pairs $x_i \leftrightarrow y_{i1}...y_{iq}$. Yes, I can generate more $x$ values, but it will not help in taking additional data into account, if I understood you correctly. – German Demidov Sep 29 '16 at 9:59
• @GermanDemidov: I fear you misunderstood my answer: I intended there to describe the probability model generating your actual/observed data as I understand from your (rather confusing) description. I did not mean to tell you to generate more data. Only after writing a completed probability model and its likelihood function can you hope to estimating the parameters of this model by an EM algorithm. – Xi'an Sep 29 '16 at 20:35
• yes, I agree, but I really do not know the underlying model. The positive integer $m$ is generated from a complex model. But I got the point - it is not possible to fit EM unless the generating model is specified... – German Demidov Sep 30 '16 at 9:38
• But can you assume that the $m$ generating model to be independent of the parameters of the mixture? In this case, it would not matter for EM... – Xi'an Sep 30 '16 at 9:55
• $m$ is dependent, I think, unfortunately. Shifted Poisson is a good idea and I will try it anyway - in this science everything that works is allowed and I did not invent anything better. I will try and write what I got. Now I calculate weights of clusters with additional data (since they are normalised to 1) and I discovered that at least EM makes less steps and have the same results for the right model and worse results for the wrong model. – German Demidov Sep 30 '16 at 17:52