# Combining Probability Density Functions

I am trying to predict the outcome of a random variable x, which is a real-valued number. In some cases I can observe another variable y1, which should approximate x. I model y1 as a Gaussian distribution with mean of 0 and an empirically estimated standard deviation, and use the probability density function of that Gaussian to predict x.

In some cases I also have a second observation y2, which I can model similarly as a Gaussian.

What is the appropriate way to combine y1 and y2 into an estimate on x? Should I add the distributions and model x as a mixture of Gaussians, or should I multiply them? Or, should I try both and pick the answer that maximizes the probability of sampling an independent set of data?

• What do you mean by "n some cases I can observe ...". Does that man that neither y1 nor y2 can be observed all the time (i.e. for all the x)? – steffen Dec 10 '10 at 11:02
• @James And what do you mean when you call y2 a "prediction"? Isn't it functioning here as a predictor? – whuber Dec 10 '10 at 16:39
• @steffen - that's correct, sometimes I have estimates of x independent of any y's. – James Thompson Dec 10 '10 at 17:24
• Do you have observed values of x at least sometimes matched with observed values of y1 and/or y2? – Firefeather Dec 10 '10 at 21:28
• @Firefeather - Yes I do. I'd like to know what's strictly right versus what's pragmatic to do - maybe modeling these as Gaussians isn't right, so adding or multiplying is the theoretical right answer but not practical. – James Thompson Dec 11 '10 at 0:54

## 1 Answer

I can tell you what I would do as a machine learner ;):

1. Creating two models $M_1,M_2$ for x using $y_1,y_2$ respectively
2. A prediction for x is calculated as the average of the predictions of $M_1$ and $M_2$

A "model" can be either a gaussian distribution (as described, if you know that x a) has a gaussian distribution and b) know that the function x to y is nearly the identity) or anything else, e.g. a simple linear regression model (if you are not sure whether there are additional factors).

BUT: If x is indeed a bimodal distribution with the two gaussians y, than the suggested approach would not make any sense. In this case I'd try a more generic approach, i.e. EM to see whether one or two gaussians are more appropriate.

It is quite hard to give a general answer to this questions, since it is not exactly clear what determines whether y1 or y2 can be observed. It could be that for both y x is missing completely at random (in this case y1 and y2 were just different random samples), but on the other hand it could be that for a certain fraction of x only y1 can be observed, but not y2, and vice versa for another fraction of x.