I have a problem where I am trying to classify data into two groups using a single parameter. The distribution of this parameter is Gaussian for two groups, so what I'm dealing with is two overlapping Gaussians. If I knew what the parameters of both Gaussians, I could just use Bayes theorem to calculate the posterior probability for each data point directly.

However, I only know the means and standard deviations, so I use what I think is a very simple Empirical Bayes method to solve for the relative magnitude of each Gaussian. It's an iterative approach, I start with a guess for the relative magnitude of each Gaussian, calculate the posterior probabilities and then sum them for each group to get an estimate of the relative magnitude of each Gaussian which I then plug back in to get a new posterior probability. I continue this until the difference in the ratio of the sums of posterior probabilities between iterations is less than 1%.

So this works well, but I don't understand why? Like why does the sum of posterior probabilities always push the solution in the right direction, and why does it converge?

I understand I could just fit two Gaussian to the data, but this approach works better than direct fitting, and I don't understand why it does. I appreciate any help, thanks.

  • $\begingroup$ Isn't this a form of expectation-maximization (EM)? $\endgroup$ – Avraham Jan 27 '15 at 23:03
  • $\begingroup$ could you please use symbols to describe precisely your model, the data and the parameters involved in your problem? As described, it does not sound like empirical Bayes but rather as an iterative way to reach maximum likelihood, what I called "prior feedback" in a 1993 paper. $\endgroup$ – Xi'an Jan 28 '15 at 7:01
  • $\begingroup$ I looked up what EM is and I guess now I realize what I'm doing is EM method for Gaussian mixture, but in my case I'm fixing standard deviation and mean so I'm just iterating on the membership weights. But because I was not using an EM method I didn't make my convergence condition a change in log-likelihood but jut a change in the sum of the ratio of membership weights. $\endgroup$ – Mark Jan 28 '15 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.