1
$\begingroup$

I have been trying unsuccessfully for a while to understand and setup an inference problem and use EP with it. I have tried to extract a simple case to highlight he issue. I hope someone will be kind enough to offer some insight into this.

I have a graphical model as shown in the image below:

A simple PGM

I am trying to estimate a parameter set $w$. Note that $y$, $x$ and $\phi$ are observed. Now, this can be written as a factor graph as follows:

Corresponding factor graph

Now, the parameter $w$ is modelled as a multivariate distribution with a mean vector specified by $m$ and has a covariance structure $\Sigma$. The idea is that these parameters need to be spatially regularised and this is enforced by this covariance structure.

Now, the likelihood can be broken down as product of individual likelihoods (I know…the factor graph should show multiple factors one for each likelihood). Anyway, the likelihood can be written as:

$$ P(y|w,x,\phi) = \prod_n^{N} p(y_n|w_n,x_n,\phi) $$ and an individual likelihood can be written as:

$$ p(y_n|w_n,x_n,\phi) = \frac{\phi}{2\pi} \exp^{-0.5 (y_n - t(x_n,w_n))\phi(y_n - t(x_n,w_n))} $$

So, this is minimising the sum of square difference between the observed $y_{i}$ and the transformed $x_{i}$ according to the estimated parameter $w$ and some transformation function $t$.

Now, I have been watching Tom Minka's lecture and doing reading on EP. However, according to the lectures I will be refining the factors in turn. So, imagine that I am refining the ith likelihood term, so I will have a product as follows:

From all the likelihood terms, I will have a term $q^{\backslash i}(y|w^{\backslash i}, x^{\backslash i}, \phi)$. So this is a product of all the approximated likelihood terms (except the ith term). Now, this has to be multiplied with the approximate prior term $q(w)$. Now $q(w)$ is a multivariate distribution with the same form as $P(w)$, a MV normal distribution. So, I am not sure how to do this step of $q(w)q^{\backslash i}(y|w^{\backslash i}, x^{\backslash i}, \phi)$.

I have a feeling that I am missing a trick where the factor graph perhaps need to be refined further but am totally at a loss on how to do it.

I would really appreciate any tips/guidance you can give me. I have been really banging my head against this for weeks.

$\endgroup$
  • $\begingroup$ First of all, why do you want to do this on your own? Have you considered using Infer.Net ? Also, the problem is a little unclear to me. So, your problem is to find the result of multiplying two Gaussian? (i.e. q(w) * q\i(y|...)) $\endgroup$ – Daniel Feb 13 '14 at 20:47
  • $\begingroup$ The problem was that my posterior is n-dimensional whereas each of the likelihood term is modelled IID and is uni-dimensional. So, the issue becomes how do you remove this term from the posterior as the division q/t does not make sense due to dimensional mismatch. As for why I am doing it myself, it is so that I can understand it in depth. $\endgroup$ – Luca Feb 13 '14 at 21:33
  • $\begingroup$ I see; But you can choose your approximating factor to be of the same dimension. Can't you? $\endgroup$ – Daniel Feb 13 '14 at 21:38
  • $\begingroup$ Yes, that is what I had missed. So, ended up defining the approximating factor with the same dimensions but just not depending on the other variables. The way I did it was to initialise the mean vector of n-dimensions with the same mean and a diagonal covariance with the same standard deviation. I am not sure if that is correct but we will see... $\endgroup$ – Luca Feb 13 '14 at 21:45
  • $\begingroup$ Also, you seem to know more about EP than I do...would you mind looking at this question I asked :-) stats.stackexchange.com/questions/86448/… $\endgroup$ – Luca Feb 13 '14 at 21:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.