I am trying to predict the outcome of a random variable x, which is a real-valued number. In some cases I can observe another variable y1, which should approximate x. I model y1 as a Gaussian distribution with mean of 0 and an empirically estimated standard deviation, and use the probability density function of that Gaussian to predict x.

In some cases I also have a second observation y2, which I can model similarly as a Gaussian.

What is the appropriate way to combine y1 and y2 into an estimate on x? Should I add the distributions and model x as a mixture of Gaussians, or should I multiply them? Or, should I try both and pick the answer that maximizes the probability of sampling an independent set of data?

  • $\begingroup$ What do you mean by "n some cases I can observe ...". Does that man that neither y1 nor y2 can be observed all the time (i.e. for all the x)? $\endgroup$ – steffen Dec 10 '10 at 11:02
  • $\begingroup$ @James And what do you mean when you call y2 a "prediction"? Isn't it functioning here as a predictor? $\endgroup$ – whuber Dec 10 '10 at 16:39
  • $\begingroup$ @steffen - that's correct, sometimes I have estimates of x independent of any y's. $\endgroup$ – James Thompson Dec 10 '10 at 17:24
  • $\begingroup$ Do you have observed values of x at least sometimes matched with observed values of y1 and/or y2? $\endgroup$ – Firefeather Dec 10 '10 at 21:28
  • $\begingroup$ @Firefeather - Yes I do. I'd like to know what's strictly right versus what's pragmatic to do - maybe modeling these as Gaussians isn't right, so adding or multiplying is the theoretical right answer but not practical. $\endgroup$ – James Thompson Dec 11 '10 at 0:54

I can tell you what I would do as a machine learner ;):

  1. Creating two models $M_1,M_2$ for x using $y_1,y_2$ respectively
  2. A prediction for x is calculated as the average of the predictions of $M_1$ and $M_2$

A "model" can be either a gaussian distribution (as described, if you know that x a) has a gaussian distribution and b) know that the function x to y is nearly the identity) or anything else, e.g. a simple linear regression model (if you are not sure whether there are additional factors).

BUT: If x is indeed a bimodal distribution with the two gaussians y, than the suggested approach would not make any sense. In this case I'd try a more generic approach, i.e. EM to see whether one or two gaussians are more appropriate.

It is quite hard to give a general answer to this questions, since it is not exactly clear what determines whether y1 or y2 can be observed. It could be that for both y x is missing completely at random (in this case y1 and y2 were just different random samples), but on the other hand it could be that for a certain fraction of x only y1 can be observed, but not y2, and vice versa for another fraction of x.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.