How to mix probability estimators of the same phenomenon? Also posted here and here.
I have the following problem:
I have N models that give me an estimation of the probability distribution function p(x) of a certain phenomenon x. Let's call them: $p_1(x),...,p_N(x)$. They come from different sources of information, so they can give different values, but they all refere to the same observable fact. Is there a formal and valid way to combine them into a single formula?
I have read about mixture distributions, are they applicable to my case?
Suppose that my models can be more or less reliable, depending on the source of information they are based on, can I also combine them giving more weight to one model rather than another?
One possible solution I have thought is to make a weighted average of all the PDFs: $p(x) = w_1*p_1(x) + ... + w_N * p_N(x)$ where $\sum\limits_{i=0}^N w_i = 1$, does it make sense?
Thanks very much for your suggestions!
PS: to be more concrete my models give me the probability distribution that a certain person is in the position (x,y) and they rely on different sources of information like the power of a received signal or some other observable fact.
 A: I think Bayesian model comparison might be what you are looking for. See for example Bishop, Chapter 3.4.
Generally speaking, given a set of $N$ models, you choose your weights to correspond to the posterior probability of each model.
$$ p(M_i | D) \propto p(M_i)p(D | M_i) $$
where $p(M_i)$ is the prior of model importance, you can assume this to be uniform, and $D$ is your data. Hence $p(D | M_i)$ is simply the likelihood of model $i$ given data.
The predictive probability for a new value $y^*$ and explanatory values $\mathbf{x}$ is then:
$$ p(y^* | \mathbf{x}, D) = \sum_{i=1}^N p(y^*|\mathbf{x},D,M_i)p(M_i | D)$$
where we use the posteriors as weighting between models.
That's the theory. Now in practice, unless you happen to be dealing with conjugated probabilities, you won't get a closed form solution for those posteriors and the above is pretty much useless.
When people fit a mixture of distributions, they usually use a mixture of Gaussians and in rare occasions a mixture of t distributions. I think the reason is simply that the mixture is fit using the EM algorithm (again, see Bishop) and Gaussians are particularly useful since their posterior is again a Gaussian and you can get all the required updates for the EM algorithm in closed form solution. And when I say "fit", they don't fit them individually, but learn the best parameters for all mixture components from the data, which is not what you are doing.
Don't worry about that for now and simply check whether you can get the posteriors for your model, or whether you don't want to fit your mixture using the EM and some well known distribution, such as the Gaussian, or t distribution in case of many outliers.
EDIT (to your comment):
So first of all, my notation: $y$ is the quantity you are trying to model, $x$ is a vector of data used to model $y$.
The data $D$ is just a tuple of vectors $x$, used to model a tuple of $y$'s. (basically think any normal dataset with a dependent variable and multiple independent ones)
OK. now, if I understand correctly you are trying to fit a complex distribution and you somehow already know the models that explain the data (perhaps because you know the generating mechanism) so you only need the mixing proportions.
You could try the following, which is guaranteed to work for Gaussians, and I don't see why it shouldn't work for other distributions too. (though there is a big caveat here!)
Calculate for each point x the likelihood that the point was generated by model $f1$ and $f2$, where the parameters are known. You end up with a matrix of likelihoods $2 x N$ where N is the number of your points. sum each row and divide by N. You should get the responsibility of each of the models for generating the data.
Use that to weight your mixture. You should also check if the resulting density integrates to 1, and if not normalize appropriately.
