Toy explanation: I have set of different cars of different colours. There can be green, blue, red, etc. cars. I have a set of classes i.e.: "The set contains blue, red and pink cars" or "The set contains only green cars". Let's assume that colour of the cars from the set can be modelled as the mix of normals with known means/variances.
I can also look at the people that are sitting in the cars. For example, there can be 10 people that wear green in the green car. It increases the evidence that this car is green. Or I can have a car that has a colour between blue and purple and 3 persons who are wearing purple inside the car. I would say that the car is purple. And I can have a car without any people inside.
I would use EM algorithm if I would deal only with the colours of cars. But how to take people that can probably sit inside the car into account? The number of people inside different cars is different so the likelihoods will be different for each point (I mean that we can not just multiply likelihoods no more, some likelihoods will be extra small just because a lot of people are in the car).
How to fit Gaussian mixture on colours taking additional data points into account? There is no problem for classification of each separate point: the number of people is constant so the likelihoods will be "comparable" between different models, i.e. "This car is green" vs. "This car is blue". How to calculate the joint likelihood of all set of cars?
In other words, I have an ordered vector of values:
$(x_{1}, x_{2}, ..., x_{n})$ and each point were generated from Gaussian with fixed mean and $\sigma$. We know the possible set of means (so we can use constrained EM algorithm).
Example of model for vector of $x$s: mixture of Gaussian components with means (0, 1.0, 2.0, ..., 5.0) and $\sigma=0.05$. Another model can be mixture of components with means (2.0, 3.0, 4.0) and the same $\sigma$. My goal is to decide which model is true.
Each component with different mean can produce additional data: $x_i \leftrightarrow \{y_1, ...\}$. So if the point $x_i$ belongs to the Gaussian mixture component with mean 2.0, it can produce 0, 1 or more additional points that "says" that the pair $x_i \leftrightarrow \{y_1, ...\}$ belong to mixture component with the mean 2.0 (the likelihood $L(x,y|\mu=2.0)$ is bigger than $L(x,y|\mu \neq 2.0)$). The problem is that the number of additional points for each $x_i$ is different (it is possible to be $x_i \leftrightarrow \{\emptyset\}$). So if we will try calculate joint likelihood of $x_i \leftrightarrow \{y_1, ...\}$ just by multiplying probabilities $p(x_i|\mu=2.0)\cdot p(y_1|\mu=2.0)\cdot \ldots \cdot p(y_m|\mu=2.0)$ we will have the values of different order for different points!
UPD: Since membership weights of any point in mixture modelling are normalised to $\sum \pi_k = 1$, can I use the additional data to estimate at least mixture proportions more accurate? It seems that they will remain the same for the case when there is no additional data and it will be only better taking additional data into account. Now I do not see a way how to take into account additional data for the final likelihood calculation, but membership weights can be adjusted, if I am not wrong.
UPD1: The best idea so far: to create a sample of equal length from cars/people using bootstrap. So $x_i \leftrightarrow \{y_1, ...y_m\}$ for all $i$. If the set of $y$ is empty, we can repeat $x_i$ $m$ times. Looks super bad, but...
