MAP estimate of posterior parameters

Question

I have a setup where the joint posterior is written as:

$$ P(w, \lambda, \phi \vert y) = P(\phi) \times P(w \vert \lambda) \times P(\lambda) \times \prod_{i=1}^{N}P(y_i \vert w_i, \phi, \lambda) $$

Now $\phi$ and $\lambda$ are modelled as Gamma distributions and the likelihood and prior on $w$ is normally distributed. So, taking advantage of the conjugate prior property, the posterior on $\lambda$ and $\phi$ will also be Gamma distributed.

So, I can derive expressions for $P(w|\lambda, \phi, y)$, $P(\lambda|w, \phi, y)$, $P(\phi|\lambda, w, y)$ because of the conjugate property. What I would like to do is estimate the parameters $w$, $\phi$ and $\lambda$. So my idea was to start with some initial initialisation of $w$, $\phi$ and $\lambda$, and then while keeping the others fixed, update one of them in turn by maximising the log-posterior probability of these derived expressions for the marginal distribution using some simple optimization routine like conjugate conjugate gradient descent.

The questions I have are the following:

• Is this the EM way of doing things or have I stated the EM algorithm? I am finding the parameters that correspond to a local maximum mode but in the discussion of EM I always read that this so called latent variables must be present in an EM formulation and I have never got my head around that? What is the difference between the method I have proposed and the EM algorithm?

• Will this approach find me the local optima corresponding to the maximum of the full joint distribution i.e. $P(w, \lambda, \phi \vert y)$. I am doing this iteratively and does the solution suffer because of this? I am guessing that this will get me the joint optimum but could not completely convince myself. These iterations, I guess, correspond to computing the partial derivatives w.r.t to each of the parameters to be estimated in turn, which is similar to optimisation of a multi variable function.

• Also, is this a reasonable way to solve the problem? I know I will only be getting a local solution but are there any other approaches to tackle this problem in this generative model setup?

Answer 1

What you are suggesting is not exactly EM. The method should work: what you are doing is in essence, coordinate descent along your parameter set. Barring the regular pathologies one can think of in coordinate descent problems, your solution should reach the local optima. This sort of method is used in Support Vector Machines.

The terminology in EM is kind of confusing. Here is a chart which is hopefully of some use.

1) The observed variables are the ones you are dealing with (w in your case). The hidden variables are those random variables which are latent because they are contrived and do not appear in your result (if you have $W|Y$ then $Y$ is the hidden variable).

2) The parameters are all the variables which determine the distribution of your parameters.

3) In connection with your problem, in EM one would update all the parameters in the E step and in the M step maximize the expected complete data log likelihood where the expectation is taken with respect to the posterior distribution.

Think of it this way, I give you some prior information, you use your data to find a posterior estimate --> "this is what the prior should be after I look at the data" and find parameter estimates using the complete data log likelihood weighted by the probability of noticing your hidden variable which is the M step.