I have observed data $D$ about a physical object described by $M$. I would like to determine the posterior distribution of $M$ given $D$, or $p(M|D)$. Now I can't infer this directly because unknown correlations in the observations. So I use parameters $\theta$ to smooth out the data $D$, reducing its effective "weight."

So now the problem to determining the posterior probabillity $p(M,\theta|D)$. Thus I need to optimize $M$ at the same time as $\theta$. I think the way to express this problem now is with the graphical model $M \rightarrow \theta \rightarrow D$. It seems that there are two linked inference problems:

  1. inferring $\theta$ from $D$:

    $$ p(\theta|D) = \frac{p(D|\theta)p(\theta)}{p(D)} $$

  2. inferring $M$ from $\theta$:

    $$ p(M|\theta) = \frac{p(\theta|M)p(M)}{p(\theta)} $$

How do I do both these inferences at the same time? Is there a simpler way to write an equation for the whole thing? It's really confusing because in (1), $\theta$ is the "model" but in (2) $\theta$ is the "data." Note that I have priors for both $M$ (keeping it physically reasonable) and $\theta$ (reducing its complexity).

Edit: My first attempt to solve this is with the "hierarchical Bayes" method: $$ \begin{align} p(M,\theta|D) &\propto p(D|M,\theta)p(M,\theta) \\ & = p(D|M,\theta)p(\theta|M)p(M) \\ & = p(D|\theta)p(\theta|M)p(M) \end{align} $$ Where the last step uses $D\perp M|\theta$ from the graphical model. While this is appealing, there's one thing missing: there should be a place for $p(\theta)$ as well as $p(M)$. That's because I can say something about the uncertainty parameters $\theta$ independent of everything - they should be as simple as possible. Maybe I need a different way to formulate the problem? Perhaps a different graphical model? Thanks for your help!

  • $\begingroup$ What distributions are known exactly other than $p(M)$ and $p(\theta)$? $\endgroup$ – rwolst Mar 4 '15 at 12:36
  • $\begingroup$ We know also $p(D|\Theta)$ and $p(\Theta|M)$. Although I'm not totally sure the difference between $p(\Theta|M)$ and $p(M|\Theta)$ in this case. $\endgroup$ – simurg Mar 4 '15 at 13:00
  • 1
    $\begingroup$ I don't see the issue with using the hierarchical Bayes method here. The marginal distribution of the $\theta$ in this case is unnecessary. $\endgroup$ – rwolst Mar 4 '15 at 13:09

Try with a Factor Graph and use Belief Propagation to get your posterior. Your Problem reminds me of Compressed Sensing, there you get the posterior p(x,s|y) {with y=Ax where y is the observed Data x is the Signal and s ist the sparsity pattern}.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.